Service Health

SUMMARY: On Friday 21 November, Google Compute Engine instances in the asia-east1 region experienced elevated packet loss for 70 minutes. We sincerely apologize for the disruption in service to these instances.

DETAILED DESCRIPTION OF IMPACT: Beginning at 00:04 PST, Compute Engine instances in the asia-east1 region experienced up to 90% packet loss. Most packet loss was mitigated by 00:31, however traffic between instances in the europe-west1 and asia-east1 regions continued to experience 60% loss until 01:10.

ROOT CAUSE: This incident was caused by an unusually large surge in traffic.

REMEDIATION AND PREVENTION: The spike in traffic was identified by Google engineers within 3 minutes of impacting other traffic. A combination of automated and manual intervention by Google engineers mitigated the majority of the impact within 45 minutes.

To most effectively handle this class of incident in the future, Google engineers are increasing the granularity of network monitoring, which will result in faster diagnosis of such issues. Google engineers are also rolling out additional automated systems to improve the way the system manages traffic when high volumes are seen.

The network problem affecting the "asia-east1" region was resolved as of 01:10 US/Pacific.

We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We will provide a more detailed analysis of this incident once we have completed our internal investigation.

The problem with network affecting "asia-east1" should be resolved as of 01:10 US/Pacific. We are continuing to monitor it to prevent re-occurrences of the issue.

We will provide an update by 04:30 US/Pacific in this forum: https://groups.google.com/forum/#!topic/gce-operations/jt_v31tnue8

We're investigating a networking issue affecting "asia-east1" region beginning on 2014-11-20 at 23:59 US/Pacific. We will provide more information shortly.