Service Health

SUMMARY: On Monday 14 July 2014, most users were unable to create or delete instances that utilize external IP addresses in all Google Compute Engine zones for a duration of up to 135 minutes. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. We are taking immediate and ongoing action to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT: The incident affected Compute Engine users who attempted to create or delete instances with external IP addresses. Beginning at 15:30 US/Pacific users experienced longer than usual instance creation times, or in many cases received errors, when creating or deleting instances. By 17:45 this issue was resolved. Existing virtual machine instances continued operating normally throughout the incident.

ROOT CAUSE: Due to growing demand, a subsystem responsible for tracking and allocating Compute Engine instance IP addresses became overloaded. As a result of this, an increasing number of requests to allocate external IP addresses for new instances, or deallocate IP addresses as part of deleting existing instances, timed out or received errors. This caused the system to retry the requests, increasing load and prevented many requests from completing.

REMEDIATION AND PREVENTION: Google engineers resolved the problem by reducing the number of tasks attempting to allocate or deallocate IP addresses. This alleviated load on the data replication layer and allowed the systems that allocate and deallocate IP addresses to complete work more rapidly, which in turn cleared the backlog, returning the service to normal operation.

To prevent this issue from recurring, Google’s engineering team has provisioned additional capacity for the system that was overloaded, and added monitoring to alert engineers when additional capacity is needed and before the system begins to overload. Google engineers are also optimizing the code which allocates IP addresses to increase its capacity, and are decreasing the number of queries required for IP address allocation.

The problem with creating new Compute Engine instances beginning at 15:30 PST should be resolved as of 17:45 PST. We apologize for any issues this may have caused you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

We are still investigating the issue with creating new Compute Engine instances. We will provide another status update by 19:45 PST.

We are still investigating the issue with creating new Compute Engine instances. 70% of the projects recovered. We will provide another status update by 18:30 PST.

We're investigating an issue with creating new Compute Engine instances beginning at 16:39 PST. We will provide more information shortly.