Network Issues Cause Interruptions; Services Now Fully Restored

On Tuesday morning, Nov. 13, Stanford experienced two network related issues that resulted in clients being unable to access some online computing services, including email. By late morning, University IT (UIT) addressed both issues and all services are now operating normally.

Online services inaccessible

Late Monday evening and into Tuesday morning, some members of the Stanford community were unable to connect to online services, including Oracle Financials, Axess, Research Administration and others that required them to log in using Stanford authentication. UIT networking engineers isolated the cause to a device on Stanford's network infrastructure. Built-in backup capabilities provided a temporary fix, but eventually the device required a restart. The restart corrected the problem and the affected online services were accessible again at approximately 8:40 a.m.

Email delayed

At approximately 7:30 a.m. Monday, some members of the Stanford community experienced delays connecting to Stanford Office 365 email using the Stanford network. (Clients connecting to Stanford email off the Stanford network could connect normally.) Microsoft received reports of similar issues from other west coast clients and isolated the issue to locations that were using CENIC's California Research and Education Network (CalREN). CENIC corrected the issue and email service returned to normal at approximately 9:45 a.m.

Services restored, monitoring continues

All services are now restored and fully operational. UIT will continue to monitor these services and methods to reduce the impact of similar issues.

We apologize for any inconvenience these service outages caused.