Service Health

SUMMARY: On May 28th 2014 from 1:42pm PDT to 2:52pm PDT approximately 12% of Google Compute Engine instances experienced network connection failures. If your service or application was affected, we apologize. We are making adjustments to our service to ensure this issue cannot occur again.

DETAILED DESCRIPTION OF IMPACT: Approximately 12% of instances, 95% confined to us-central1-a, experienced persistent failures of network connections to some or all remote hosts at times between 1:42pm and 2:52pm PDT. The instances themselves and their disks were not harmed and resumed normal behavior after the incident.

ROOT CAUSE: A routine infrastructure upgrade resulted in a number of instances needing to perform a lookup of network addresses and temporarily increased the workload involved in serving these requests. The service that processes these requests was not provisioned sufficiently to handle the increase workload, resulting in the failure of network connections requiring an address lookup.

REMEDIATION AND PREVENTION: Google engineers are increasing the capacity of the lookup service to handle future demand spikes. The lookups are being changed to handle transient failure resiliently. Finally, the lookup service will be isolated between different GCE zones to prevent failures in one zone from having an impact on other zones.

The Google Compute Engine network problem that started on May 28th 1:50pm PDT is resolved as of 3:45pm PDT. We apologize for any issues this may have caused you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are still investigating the issue with Google Compute Engine networking. Networking is currently operating as normal. We will provide another status update by May 28, 5pm.

Connectivity to Google Compute Engine instances is largely restored. We will post another status update by May 28th 4pm.

We are still investigating the issue with Google Compute Engine network connectivity. We will provide another status update by May 28th, 3:30pm Pacific Time.

We are currently experiencing an issue with Google Compute Engine and some users are experiencing loss of network connectivity to their instances. For everyone who is affected, we apologize for any inconvenience you may be experiencing.

We will provide an update by May 28th 3:00pm Pacific Time with current details.