SUMMARY:

On Sunday 12th April 2015, Google Compute Engine instances in us-central1 and asia-east1 experienced intermittent 0 to 4 percent packet loss on outgoing external traffic for an overall duration of 6 hours. This was followed by a peak of up to 20 percent packet loss for a duration of 13 minutes. We apologize to our customers who were affected by this issue, and are working to address the factors that allowed this to happen.

DETAILED DESCRIPTION OF IMPACT:

On Sunday 12th April 2015 from 16:26 to 22:24 PDT, Compute Engine experienced packet loss totalling 0 to 4 percent of traffic to external addresses. Individual instances could have experienced up to 100 percent loss intermittently. The duration of packet loss was zone-specific. The issue began in us-central1-a at 16:26 and lasted until 22:24. The zones us-central1-b, us-central1-f, and asia-east1-a experienced loss between 19:20 and 20:20.

Total loss ranged from 2 to 4 percent between 19:27 and 20:03, and 2 to 20 percent between 22:20 and 22:40. Outside these time periods, loss was under 2 percent of total egress traffic.

ROOT CAUSE:

An abrupt increase in traffic from several large projects triggered Compute Engine's traffic shaping mechanisms. These mechanisms had an unintended spillover effect on other projects, causing some packet loss even for uninvolved projects.

REMEDIATION AND PREVENTION:

Google engineers increased the traffic capacity dedicated to Compute Engine to address the usage increase and eliminate packet loss. Google’s engineers are also making code changes to improve the fidelity and speed of response of the traffic shaping mechanism, and eliminate packet loss for uninvolved projects in future traffic shaping events.

The problem with Google Compute Engine Networking is resolved as of 2015-04-12 22:40 US/Pacific. We verified there weren't any replicas of this issue since that time.

The problem with Google Compute Engine Networking should be resolved as of 22:40 US/Pacific. Our engineers identified the root cause of the issues and deployed a fix that is preventing packet loss since that time. We will continue to monitor the situation and post an all clear around 2015-04-13 00:00 US/Pacific.

We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support.

We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are still investigating the issue with Google Compute Engine Networking performance. As of this moment, the issue is isolated to the us-central1-a region. We will provide another status update by 2015-04-12 23:00 US/Pacific.

We are still investigating the issue with Google Compute Engine Networking performance. We will provide another status update by 2015-04-12 22:00 US/Pacific.

We are currently experiencing an issue with Google Compute Engine Networking performance and some users are seeing higher packet loss rates. We would like to apologise for any inconvenience caused and reassure you that our reliability staff is currently handling this issue as maximum priority.

This issue started on us-central1-a and us-central1-f, but has progressed to affect all zones.

We will provide an update by 2015-04-12 21:00 US/Pacific with current details.

Correct incident start time: 2015-04-12 17:00 US/Pacific.

We're investigating an issue with Google Compute Engine Networking performance beginning at around 2015-03-12 17:00 US/Pacific. We will provide more information shortly.