Service Health

SUMMARY: On Friday 19 December 2014, all Google Compute Engine instances in us-central1-a experienced elevated packet loss for a duration of 83 minutes. We apologize for the disruption.

DETAILED DESCRIPTION OF IMPACT: Beginning at 09:58 PST, 10% of packets to and from instances in us-central1-a were dropped. At 10:38, the drop rate increased to 60% for the next 32 minutes. At 11:10, the drop rate returned to 10% for another 10 minutes, before the end of the outage at 11:21.

ROOT CAUSE: The incident was caused by interference between two components of the GCE networking infrastructure during a routine software deployment.The interference began at 09:58, redirecting traffic through a lower capacity path resulting in 10% packet loss to and from instances in us-central1-a. At 10:38 the second component began rolling out, redirecting traffic to the original path with reduced capacity and increasing packet loss to 60%.

REMEDIATION AND PREVENTION: Google engineers were notified of the issue by monitoring at 10:23. Google engineers started rolling back to the previous version 11 minutes after the packet loss increased. When the second component reverted, traffic loss was back to 10%. Finally, the engineers completed the rollback, resulting in the end of the outage.

Google engineers are eliminating the race condition in the rollout process, as well as updating related documentation and monitoring to enable faster diagnosis and remediation.

The problem with Google Compute Engine should be resolved as of 12:10pm PST. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are still investigating the issue with packet loss to/from VMs in us-central1-a. We will provide another status update by 12:15pm PST.

We are still investigating the issue with packet loss to/from VMs in us-central1-a. We will provide another status update by 12:00pm PST.

We are still investigating the issue with packet loss to and from VM instances in us-central1-a. We will provide another status update by 11:45 PST.