IE BUMPER IE BUMPER IE BUMPER University Applications > Office of IT Home Page (ServiceNow) > Issues with BYU Services IE BUMPER

Issues with BYU Services

See: Official OIT Statement

See: Daily Universe Article

Update 7/10/12 9:54a.m. - Recovery is still continuing on remaining services still affected and will be for quite some time. Most services are now back up and running. Please talk with your CSR if you have questions on when your services will be restored. As new information on the recovery is obtained, we will post new notices as appropriate. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/26/12 7:45a.m. - Recovery continues on any remaining services still affected. If you have any questions or concerns about a product or service, please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/21/12 11:17 am - The issues yesterday afternoon of logging into Learning Suite and other BYU Resources should now be solved.

We are continuing the work migrating resources to new storage. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/20/12 4:21 pm - Some users are unable to get into the Learning Suite and other various BYU Resources. Engineers are currently working on a fix for this. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/20/12 1:54 pm - Users should now be able to log in to BYU resources. Some may need to reset their password again if they tried when the system was having difficulties.

Network drives are still experiencing some difficulties that are being worked on. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/20/12 1:48 pm - Users should be able to authenticate to the BYU Secure wireless network now.

 Some users are still unable to log in to BYU resources even after resetting their Net ID password. This is currently being worked on.

Network drives are still experiencing some difficulties that are being worked on. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/20/12 1:29 pm - Some users are unable to log in to BYU resources even after resetting their Net ID password. We are also seeing issues where users are unable to authenticate to BYU Secure. Also, some users cannot get to their resources on network drives. Engineers are working on resolving all of these issues. If you have any questions or concerns please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/20/12 8:37 am - Work is still continuing on migrating resources to new storage. We are continuing to help restore services. If you see lingering issues please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/15/12 11:35 am - The DATA1 shared drive is now back online.  If you continue to have problems with your shared drive or have any other concerns, please let the OIT Service Desk know at 801-422-4000 or it@byu.edu.

Update 6/15/12 10:47 am - The DATA1 shared drive is inaccessible this morning. If you see these issues please contact the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/14/12 2:30 pm  - All email should be functioning normally now.  If you are continuing to see issues, please contact the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/14/12 11:30 am - We are experiencing a problem with one of the email servers.  Approximately 1/8th of Exchange email users is affected and can not connect to email at this time.

Update 6/12/12 9:44a.m.: The following services may still be having issues: Life Sciences resources, FHSS resources, LDS Philanthropies resources, Fine Arts resources, Faculty Center resources, and visualarts.byu.edu.
Work still continues on migrating resources to new storage. We are continuing to help restore services. 

Update 6/13/12 3:20p.m.: We are continuing to migrate resources to new storage and are continuing to help restore services. If you are still having issues with any services please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.


Over the weekend and last night, Engineers ran additional restores and merges to affected email accounts. These fixes should have restored old messages, in some cases, even messages that users had already deleted. If you continue to see issues with email, please contact the Service Desk at 801-422-4000 or it@byu.edu and we will work to resolve them on a case-by-case basis.

If you are still having issues with any services please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/11/12 12:50 p.m.: Learning Suite Exams were down for part of Saturday, and also this morning.  Service has been restored as of 10a.m. today. 

We are continuing to migrate resources to new storage and are continuing to help restore services. 

If you are still having issues with any services please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.


Update 6/8/12 10:20a.m.: The Independent Study website should be experiencing normal load now. 

We are continuing to migrate resources to new storage. Currently we have 10 VMs that are powered on that still need to be moved. We believe there are less than 50 VMs that are powered off and need to be moved. We have around 200 VMs left to recover.

We are working to get the next level of recovery done for Family, Home and Social Sciences and Life Sciences.

If you are still having issues with any services please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu.

Update 6/7/12 3:15p.m.: The Independent Study website has a high volume of individuals attempting to register for courses. Some may be asked to wait on the error page they receive without refreshing the page until re-directed to the website when resources are available. If you need clarification or see issues with other services, please contact the Office of IT Service Desk at 801-422-4000 or it@byu.edu. 

Update 6/7/2012 2:30 p.m.: Services that are still unavailable or experiencing difficulties are on the servers that were most severely impacted by the failure.  Engineers and System Admins are working full time to move these servers onto stable infrastructure and get them recovered.  In some cases, they may need to rebuild the base system and then recover data from backups.  This work is expected to take many more days to address every single service that is affected.

Update 6/7/12 8:50a.m.: DATA-1 shared drives are now back up and running. Users should be able to access their shared drives now and the slowness issues should be fixed. If you have any questions or concerns please contact the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/6/12 4:50p.m.: Controllers for some servers that have not yet been migrated to the new storage were locked up. Those controllers have all been rebooted, which restored control to most servers associated with those hosts. Various campus services were affected during this reboot activity. A few servers came back in an inconsistent state or with file-store problems. Those servers are being addressed by campus CSRs and OIT Engineers. Migration to the new storage continues.

Engineers are continuing to work on the DATA-1 shared drive. This will affect some users who are trying to access data on shared drives. Please report any concerns to the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/6/12 1:21p.m.: Engineers are continuing work on the DATA-1 shared drive.  From 1:07 to 1:13 pm, there was a 6 minute core network outage.  Network connectivity have been restored. You may need to log into any sites or programs that you had up previous to the outage. Please report any lingering issues to the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/6/12 8:30a.m.: One shared drive is down this morning as work is completed on the drive. Any shared drive that connects to BYU-WORK-DATA-1.byu.edu will not be accessible this morning. Users in the ASB, Faculty Center, ITB, and MB will be affected. 

Update 6/5/12 3:45p.m: For those who have been experiencing issues with their Global Address Book in Exchange email, a solution has been deployed. This will correct this issue when the systems sync over-night and should be corrected by tomorrow. If you would like to fix this sooner you can do it manually by performing the following steps:

Outlook 2010
  1. Click on the Send/Receive tab.
  2. Click Send/Receive Groups.
  3. Click Download Address Book.
  4. Click OK in the box that comes up.
Outlook 2007
  1. Click Tools
  2. Click Send/Recieve
  3. Click Download Address Book
  4. Click OK in the box that comes up.
If you have any questions or concerns please contact the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/5/12 1:47p.m.: We have heard reports of users disappearing from the Global Address Book. We are currently working on a fix for this. Please contact the Service Desk at 801-422-4000 or it@byu.edu if you have any questions or concerns.

Update 6/5/12 11:32a.m.: Although OIT has now stabilized the infrastructure there is still much to do to restore all the services.  We will continue working with CSRs across campus to complete this work.
Here is a summary of all of the lingering email issues:
  • Some users are missing email from May28th and 29th and these are unrecoverable.
  • Some users have reported missing emails from the evening of June 1st to the morning of June 3rd. We are not sure what caused this but we believe that the senders received a rejection notice that their mail was not delivered. This mail is unrecoverable. Users can contact the Service Desk at 801-422-4000 or it@byu.edu and the Service Desk can identify potentially missing emails for concerned users.
  • Some users are missing everything in their mailbox that is older than May 29th. The old data can be recovered up through May 27th via tape backup or from the local cache (ost) if it is available. We are trying to find a way to identify all mailboxes with this issue.
  • Some users are still being prompted to use a temporary database or work offline. If you have verified that you have all of your data prior to May 29th, then you should follow the directions found HERE. If you are missing data then you should work with your CSR to import the cached data into your exchange account. Once everything is in Exchange you can follow the link above to get rid of the star-tup message.

If you need assistance with anything please contact the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/5/12 7:43a.m.: From a previous update: Y-Expense/Purchasing, Student Ratings, scheduling.byu.edu, dining.byu.edu, parking.byu.edu, housing.byu.edu, admissions.byu.edu, wellness.byu.edu (no login), yserve.byu.edu, MTC Teacher Application, learningoutcomes.byu.edu, and Group Access Admin are up and accessible to all.

Update 6/4/12 5:49pm:  Progress has been good on restoring systems and moving to the new storage system. Many individual email issues have also been resolved. Please report any issues to the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/4/12 5:00pm: Work continues on all affected services.  Please report any issues to the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/4/12 4:00pm: Efforts are continuing to restore all services. Please report any issues to the Service Desk at 801-422-4000 or it@byu.edu.

Update 6/4/12 3:07p.m.: Work continues on all affected services.  Please report any issues to the Service Desk at 801-422-4000 or it@byu.edu

Update 6/4/12 2:20p.m.: Please continue to report all email issues to the Service Desk at 801-422-4000 or it@byu.edu. 

Update 6/4/12 1:15p.m.: Efforts continue to restore all services. Please notify the Service Desk at 801-422-4000 or it@byu.edu if you experience difficulties with any BYU services.

Update 6/4/12 12:02pm: Engineers are currently still working to get everything back up and running. A few things that have been reported are that some people are missing emails, some people are having email slowness or cannot connect to their email. Many other campus resources are also still experiencing difficulties. If you notice things broken please contact the Service Desk at 801-422-4000 or it@byu.edu. Please do not assume that an issue has already been reported.  Help us make sure we have everything documented that needs fixed.

Update 6/4/12 11:00 am: Engineers are currently still working to get everything back up and running. Remember to please contact the Service Desk at 801-422-4000 or it@byu.edu. Please do not assume that an issue has already been reported.  Help us make sure we have everything documented that needs fixed.

Update 6/4/12 10:12 am: Engineers are currently still working to get everything back up and running. If you notice things broken please contact the Service Desk at 801-422-4000 or it@byu.edu. Please do not assume that an issue has already been reported.  Help us make sure we have everything documented that needs fixed.

Update 6/4 8:40 a.m.: We are recommending that individuals use mail.byu.edu if you are missing email to verify that it is not a client issue.  Please work with your CSR's if you are having problems with Outlook or other email to make sure they are configure corrrectly.

Update 6/4 7:00 a.m.: All email is up

Update 6/4 1:47 a.m.: All email databases have been copied to the new storage. Engineers are still checking data for errors on the database.

Update 6/3 9:34 p.m.:  All email databases except one have completed copying to the new storage.  The final database is projected to finish at about 1:00 AM.  Engineers are checking data for errors on the last database that just finished copying and then they will bring it online.  Learning Suite exams are still down, but other Learning Suite components should be working.  Campus CSRs are invited to ITB 1010 tomorrow, 6/4 at 9:00 AM for a detailed status update and collaborative help with rebuilding and restoring their servers.

Update 6/3 4:38 p.m.:  One email server remains to be restored.  It is in the process of being moved to the new storage.  It has a very large database and will still take several hours to finish copying to the new storage.  We hope to have email for all users back by start of business on Monday.  Storage and Server Engineers and Operations staff are working with campus CSRs to restore, rebuild and recover campus and department servers and data.  Testing on byu.edu, my.byu.edu, and Learning Suite is underway to check for any problems or inconsistencies.  If you are seeing errors in systems you use, please report it to the Service Desk at 422-4000.  Please don't assume we already know about it.

Update 6/3 1:31 p.m.:  The majority of users should now be able to access their BYU email, and both send and receive messages.  OIT is continuing to resolve the remaining issues with email.  

Update 6/3 7:30 a.m.  Progress on email restore continues to be made.  As work progresses, more and more users will see email start delivering to their inboxes.  It may take a while for backed-up messages to come through.  Gateway servers have been working throughout the weekend and all incoming email has been retained and will be delivered to inboxes.

Update 6/3 1:18 a.m.: eMail restores continue to progress. Repeating failures of disk storage has resulted in other services going up and down.  Engineers and Operations continue working to move services onto stable equipment and restore services.

Update 6/2 9.28 p.m.: Database Engineers are going to work on one of the databases related to Informatica.

Update 6/2 9.13 p.m.:
OIT Engineers are working on Learning Suite, there might experience some outage in service. Engineers are working towards stability of the service.

Update 6/2 6.42 p.m.: Learning outcomes in Learning Suite is having errors. Also, message through message tab in the learning suite is not responding. OIT is working towards the stability of service.

Update 6/2 6:29 p.m: Engineers have restored some of the e-mail account but not all. Restored e-mail accounts might see some of their e-mail missing for now but they will be restored by engineers soon.

Update 6/2 2:32 p.m.: Backups is now being transferred to more stable servers.  Once this process is complete, engineers will work to bring email back online.

Update 6/2 1:22 p.m.:
OIT is still working on a backups solution.  Once that is in place, we will be able to address email.  As always, please direct any questions to 801-422-4000, or it@byu.edu

Update 6/2 12:06 p.m.:
OIT is now looking for a temporary backup substitute that will allow engineers to restore functionality to email and other OIT services.

Update 6/2 11:04 a.m.:
Email is still down.  OIT is working with a third party vendor to determine the best course of action to take. VPN is still stable, but please report any other service outages to the Office of Information Technology.

Update 6/2 10:35 a.m.:
OIT has run into some more storage problems that is delaying the resolution of the Email problems.  VPN log in functionality went down around 9:45, but engineers have a fix in place as of 10:15.  As always please contact the Office of Information Technology at 801-422-4000 or it@byu.edu to report any additional problems.

Update 6/2 8:50 a.m.:
Email will begin coming back online for approximately 1/8th of all users. Email service will be restored gradually throughout the day as the servers that host them become available. 

Update 6/2 8:17 a.m.: Engineers are performing scheduled checks on byu email services.  Email service will continue to be interrupted while they perform this maintenance.  We will continue providing updates to the work progress as they come in.

Update 6/2 1:11 a.m.: After collaborating with vendor, OIT Engineers brought the BYU Homepage, my.byu.edu and other services up and running. They had identified the root cause which was due to the failed drive and they are going to continue working on Saturday morning. Currently, all the affected services seem to functioning normal. Please contact Office of Information Technology at 801-422-400 or it@byu.edu if experience any difficulties.

Update6/1 11:22p.m.: To make the affected storage stable, OIT Engineers are collaborating to vendor and during this course of action, BYU Homepage, my.byu.edu and other applications might have short outage.

Update 6/1 11:14p.m.: As part of the planned maintenance, Email Engineers have started working on database restore for email servers.

Update 6/1 9:31p.m.: BYU Home page is up and running. OIT Engineers are still working on to bring other applications up and running.

Update 6/1 9:02p.m.: OIT Engineers are currently working with vendors to make the storage device stable and bring all the applications up and running.

Update 6/1 7:43p.m.: Due to failure of storage device, currently BYU Homepage, Route Y applications, PeopleSoft applications and other BYU applications are not accessible. Engineers are working on to bring the applications back up.

Update 6/1 6:00p.m.: Email Engineers are still working on database on exchange mail box servers.

Update 6/1 5:04p.m.: Email Engineers are currently working on database on exchange mail box servers.

Update 6/1 5:00p.m.: Email checks are about to commence. Email will be unavailable for periods of varying duration and frequency tonight and throughout the weekend. All messages will be delivered once email accounts are back online, should any emails be sent while an email account is temporarily unavailable. 

Update 6/1 4:01pm: All current outages are still occurring and being worked on. Please contact the Office of Information Technology at 801-422-4000 or it@byu.edu if you are experiencing difficulties.

Update 6/1 3:02p.m.: Beginning today (Friday) at 5:00p.m. and continuing throughout the weekend, OIT engineers will perform checks intended to ensure full functionality to Premium Email accounts on all servers.  This will result in periods during which email will be unavailable. These periods may be of varying duration and frequency. While email will be temporarily unavailable, all messages will be delivered once it is brought back on-line. 

Update 6/1 10:24am: All of the following are still experiencing difficulties: Premium Email, Student Ratings, Importing from Blackboard into the Learning Suite, Voicemail, Y-Expense, scheduling.byu.edu, Life Sciences resources, dining.byu.edu, housing.byu.edu, admissions.byu.edu, wellness.byu.edu, FHSS resources, LDS Philanthropies U drive and P drive, Counselling and Career Center U drive, IPTV, Fine Arts resources, Student Life resources, yserve.byu.edu, Active Directory, MTC Teacher Application, Learning Outcomes, Digital Dialog, Faculty Center S drive, visualarts.byu.edu, Student Financial Services S drive, internationalservices.byu.edu, BOB Portlet, Group Access Admin.

Update 6/1 9:39am: All email databases have been restored and merged, but some individuals have been experiencing more issues with their email accounts. For some users mail.byu.edu is still working. Another thing to try is to reset the Outlook cache. Instructions on how to do this can be found here

Update 6/1 8:46a.m.: Gradebook service has been restored.  All email databases have been restored and merged at this point, and all users should see their previous messages and new messages. 

Update 6/1 2:24am: Merging of the new messages to the old had been completed.

Update 6/1 1:31am:
Change to add new disk array on FalconStore has been completed.


Update 5/31 11:32pm:
Restore for the e-mail server has been completed successfully. Merging of new messages to the old has started.

Update 5/31 8:36pm: Database restore for the e-mail servers has started.

Update 5/31 5:43pm:
The Library website (lib.byu.edu) is now up and running.


Update 5/31  5:29pm: Email users who are still missing their older email messages will not have access to any email between 8:00 pm and midnight tonight. During that time we will be restoring the database their email is using. From midnight to about 6:00 am tomorrow, those users will only see their old messages (prior to 5/27). By 6:00 am their newer messages will be merged back into their email and their email restore will be complete. 

Email users who have had their email restored may be still seeing a warning about using a temporary mailbox. Instructions to clear this warning are available here. They may also be missing rules and calendar assignments. Those will need to be re-created.

Update 5/31 5:18p.m.: The Library website (lib.byu.edu) is currently unavailable. Work is being done to restore service.

Update 5/31 2:41p.m.: Efforts continue to restore all services.  If you have any questions about specific services, please call the OIT Service Desk at 801-422-4000 or email it@byu.edu.

Update 5/31 12:32p.m.: Gradebook service is being restored and is partially up right now. That will continue to be worked on. Student Ratings currently will not allow students to submit ratings, and is also being worked on. 

Update 5/31: 10:32am: The Traffic Office and ID Center are having issues with some of their applications.

Update 5/31 8:26a.m.: Email database issues are still being worked on and will be resolved as quickly as possible. Gradebook.byu.edu and learningsuite.byu.edu are both inaccessible this morning and are being worked on. 

Update 5/30 5:15p.m.: Student Ratings has been brought back up and is functioning as normal. 

Update 5/30 3:17p.m.: Learning Suite Exams have been brought back online.  Because of data issues, exams were restored to the state they were in at 11:32a.m. yesterday (5/29/12).  Any exams taken between roughly 11:30a.m. and 3:30p.m. yesterday may be lost and will need to be re-taken.  If a faculty member wishes to re-open a test that has already closed, please contact OIT at 422-4000 or it@byu.edu.  If you see other errors on the Exams tab, please notify us.  

Update 5/30 1:27pm: Business Objects is also experiencing difficulties.

Update 5/30 12:43pm: The following are still having issues to the best of our knowledge: Academic VP Office shared R drive, Active Directory, admissions.byu.edu, Archaeology R drive, College of Fine Arts resources, Premium Email accounts, Faculty Center S drive, Family Home and Social Sciences resources, Graduate Studies resources, Parts of the Human Resources and Payroll site, Learning Outcomes, Center for Teaching and Learning resources, Learning Suite, Life Sciences resources, MTC Teacher application, Student Financial Services S drive, scheduling.byu.edu, shasta.byu.edu, Signature Card, Student Life resources, Student Ratings, Voicemail, Y-Cloud, International Studies, Y-Expense, dining.byu.edu, housing.byu.edu, marriottschool.byu.edu, wellness.byu.edu, User Management System, LDS Philanthropies U drive and P drive, Counseling and Career Center U drive, IPTV, Fine Arts Department resources, visualarts.byu.edu.

Update 5/30 11:26a.m.: Another email database has been fully restored and more affected users should see their previous emails, calendar items, and contacts. 

Update 5/30 10:44a.m.: The Library website and graduatestudies.byu.edu are accessible again. All other issues are still being worked on. 

Update 5/30 8:21a.m.: One of the email databases has been fully restored and some of the affected users will now see emails previous to 5/27. Further work on other email databases is still underway, and will go on until resolved.  A relatively small number of email users will likely have lost new email received and sent between approximately 8:00p.m. 5/27 and 8:00p.m. on 5/29. 

Update 5/30 7:45a.m.: Email databases are still being worked on and will be restored as soon as possible. Due to the large amount of data to be recovered, this effort to restore service to all users will take some time. Users who were affected yesterday and the day before may start to see messages, contacts, and calendar items reappearing throughout the few days.   Also, Learning Suite Exams are still experiencing issues. 

Update 5/29 10:15p.m.: Email databases will be down for several hours tonight while they are repaired. 

Update 5/29 8:30p.m.: Some of the issues with Learning Suite Exams have been resolved, some issues are still being worked on.

Update 5/29 4:35pm: Learning Suite Exams are having issues.

Update 5/29 3:52pm: Ecclesiastical endorsements are now working. CSRs are unable to add new users or edit those in Active Directory.

Update 5/29 3:07pm: Additional services that are experiencing difficulties: HR/Payroll benefits page, Faculty Center S drive.

Update 5/29 2:29pm: Additional services that are experiencing difficulties: Academic VP R drive.

Update 5/29 2:23pm: Additional services that are experiencing difficulties are the MTC teacher application and Learning Outcomes.

Update 5/29 1:35pm: Additional services that are experiencing difficulties are the Psychology department R drive and relate-institute.byu.edu.

Update 5/29 12:27pm: VPN is now working. Additional services that are experiencing issues are Student Ratings, Y-Expense, Life Sciences resources, saas.byu.edu, dining.byu.edu, housing.byu.edu.

Update 5/29 11:16am: Additional resources that are affected or broken include: Parts of the ecclesiastical endorsement page, admissions.byu.edu, graduatestudies.byu.edu, anatomylab.byu.edu, marriottschool.byu.edu, vpn, the library website, LDS philanthropies U drive, Counseling and Career Center U drive, IPTV, Fine Arts Department resources, Economics department R drive, Student Life resources.

Update 5/29 11:11 am: Site licenses for Mathematica, Matlab, SPSS, Amos, and Maple are now working on TEC Podiums and Open Access Labs.

Update 5/29 11:02am: Mathematica, Matlab, SPSS, Amos, and Maple are not functioning on TEC Podiums and Open Access Computer Labs.

Update 5/29 10:36am: SOA registry is now working.

Update 5/29 10:00 am: Some additional resources that are having issues are the SOA registry and voicemail passwords. 

Update 5/29 7:30 am: We are still having difficulties with Exchange which affects email.  For some, email is working fine, for others it is not.  If you get messages indicating that your email has been redirected to another location or any other kind of disruption with your email it will be best for you to use  -  mail.byu.edu  -  for the time being.  It may take several days to get all the email restored and messages about that will be communicated through your CSR.  Some departmental servers have also been affected and your CSR is aware and working on the issue if that is the case.

If you require additional support or information, you can contact the Office of Information Technology at 801-422-4000 or email at it@byu.edu.
IE BUMPER



Feedback:
:
Was this article helpful?
YesNo
Not rated
IE BUMPER
Related Articles