Starting at approximately 2:00am on Wednesday, December 10, the student email server running Microsoft Exchange 2003 experienced a system crash. We were able to determine that the cause of the failure was that our McAfee Anti-Virus scanning system detected a virus in an essential Exchange log file and deleted the file while Exchange was still active. This caused the database to fail and prevented it from coming back online.
Unfortunately, after contacting McAfee support and attempting to restore the log file that had been removed, this did not allow us to reactivate the Exchange database. It was determined that the database was damaged by the failure and claimed to be missing two log files that did not exist. We then contacted Microsoft support shortly after noon on Wednesday and attempted to restore the database from a backup that was performed on Tuesday night. Because there was still damage to the Exchange system, we then were required to perform what is known as a "hard repair", defragmentation and then a low-level integrity check to fix any errors and make sure there was no longer any corruption on the system.
Due to the large size of this database, this recovery process took nearly 36 hours to complete. We were able to successfully repair the Exchange database shortly after midnight on Friday, December 12. Student email should be fully operational at this time.
We sincerely apologize for the inconveniences this has caused our students, faculty and staff. Please be assured that we did everything we could possibly do to get service restored, in a timely fashion, with a minimal loss of stored messages. We realize that with finals in progress, this was a very inopportune time for an outage of any kind.
Our technicians worked around the clock, from the time the problem was discovered to now, and thankfully we were able to complete the system recovery. We have since made adjustments to our anti-virus scanning systems in an attempt prevent this type of system crash from happening in the future.