Service Health

We are investigating an issue with 502 errors from HTTP(S) Load Balancing. We will provide more information by 11:05 US/Pacific.

SUMMARY: On Wednesday 16 July 2014, for a period of 75 minutes newly created Google Compute Engine instances were unable to accept inbound network connections for up to 15 minutes after their creation. If you were impacted, we apologize — this is not the reliability we strive to offer, and we have taken immediate steps to improve the platform.

DETAILED DESCRIPTION OF IMPACT: The incident affected Compute Engine users who created new instances. Beginning at 13:00 US/Pacific there was a period after instance creation during which the instance was unable to accept inbound network connections. By 14:15 this issue was resolved; all existing incidents were able to accept connections and new instances were once again able to accept connections immediately after creation. Existing virtual machine instances continued to operate normally throughout the incident.

ROOT CAUSE: The portion of the system responsible for allocating Compute Engine instance IP addresses became overloaded, preventing the system which configures software defined network routes from reading the necessary information to configure the network fabric.

REMEDIATION AND PREVENTION: To resolve the immediate issue, Google engineers redirected traffic to datacenters that were performing better, reducing the replication load and resolving the issue. Afterward, Google engineers optimized the tasks which read from the IP address management system to increase future capacity. Finally, Google engineers are improving the monitors around this system in order to alert engineers before high load on the system becomes an issue.

The problem with network connectivity to newly created Compute Engine instances should be resolved as of 14:15 US/Pacific. We apologize for any issues this may have caused you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We're investigating an issue with network connectivity to newly created Compute Engine instances beginning at approximately 13:00 US/Pacific time. We will provide more information shortly.