Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Compute Engine

Issue with booting up newly created instances and packet loss in asia-east1-a

Incident began at 2015-04-25 23:25 and ended at 2015-04-26 02:14 (all times are US/Pacific).

Date Time Description
28 Apr 2015 11:46 PDT

SUMMARY:

From Saturday 25 April 2015 23:15 PDT to Sunday 26 April 2015 02:14 PDT, 7.4% of Google Compute Engine (GCE) instances in the asia-east1-a zone experienced networking issues. Affected instances were those either created or live migrated during the incident, and were subsequently unable to communicate on the network. Google Engineers resolved the network issues on Sunday 26 April 2015 at 02:14. We apologize to the affected customers and are implementing preventative measures to ensure similar incidents do not occur in the future.

DETAILED IMPACT:

Beginning on Saturday 25 April 2015 at 23:15, the GCE network control plane stopped propagating network configuration changes to the instance provisioning component. As a result, any instances that were created or live migrated during this time were unable to obtain valid network configuration and thus could not communicate on the network. Overall, the outgoing network traffic in the asia-east1-a zone was 25% lower than expected. Instances in other zones were unaffected. During the incident, network and HTTP load balancing correctly marked the affected instances as unhealthy and routed requests to instances in other zones if they were available.

ROOT CAUSE:

During a scheduled maintenance event in the asia-east1 region, a latent configuration issue resulted in a network control plane component failing to restart correctly. Automated monitoring alerted Google Engineers to the issue, and after investigation they discovered and corrected the configuration problem allowing the network control plane to start and to begin propagating network configuration.

REMEDIATION AND PREVENTION:

To prevent similar incidents, Google Engineers are currently auditing the configuration of all critical GCE components to ensure they can migrate successfully. In addition, Google Engineers are implementing more aggressive monitoring to ensure more rapid remediation of issues.
27 Apr 2015 03:19 PDT

We have experienced a problem with Google Compute Engine Networking in the asia-east1-a zone from 2015-04-25 23:25 US/Pacific to 2015-04-26 02:14 US/Pacific, which manifested itself as follows: newly created (or rebooted) VMs in asia-east1-a were not able to boot successfully. Additionally, existing VMs in that zone saw some packet loss between VMs and to the Internet. Other Compute Engine zones were unaffected.

The issue was fully resolved as of Sunday, 2015-04-26 02:14 US/Pacific. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. We will provide a more detailed analysis of this incident once we have completed our internal investigation.