Service Health

SUMMARY:

On Wednesday 8 April 2015, 15% of requests to the Beta Compute Engine Instance Groups and Instance Group Manager APIs failed for a duration of 5 hours. Affected projects experienced issues accessing Compute Engine pages in the Developers Console during the outage. Not all projects were impacted. If your user experience was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT:

On Wednesday 8 April 2015 from 11:37 to 16:35 PDT the Beta Compute Engine Instance Groups API and Instance Group Manager API returned error code 500 responses for 15% of the calls made. The Compute Engine Developers Console (http://console.developers.google.com/) pages for VM Instances, HTTP Load Balancing, and Metadata also failed for those projects due to reliance on those APIs. Users in affected projects trying to access these pages were presented with the message ‘Service internal error occurred.". Error code: "internalError”’. The gcloud compute instance list command was unaffected, but gcloud commands using the Instance Groups and Instance Group Manager APIs were also affected.

ROOT CAUSE:

A routine software upgrade to the Instance Groups and Instance Group Manager APIs increased the resource consumption of the backend servicing these APIs. As projects used the API, the backend exhausted its internal quota and was unable to service requests from new projects. Those projects whose requests succeeded before the quota had exhausted continued to use the API successfully.

REMEDIATION AND PREVENTION:

Google engineers were automatically alerted that the Instance Groups and Instance Group Manager backends were approaching an internal quota within an hour from the start of the incident. Google engineers began to revert the change that led to the increased resource consumption, and simultaneously added capacity to the service to allow it to serve new projects. The increase in quota allowed requests to the API and the Developers Console function correctly until the rollback was complete.

To prevent similar issues occurring in future Google engineers are rolling out improvements to monitoring, alerting and testing procedures. In addition, Google engineers are increasing the resilience of the Developers Console by isolating issues in backend services and APIs.

The problem with the Developers Console should be resolved as of 16:35 PDT. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We are still investigating the issue with the Developers Console where some users are seeing errors when browsing the Compute Engine related pages of the Developers Console. The gcloud command line continues to function as normal. We will provide another status update by 17:00 PDT.

We are still investigating the issue with the Developers Console where some users are seeing errors when browsing the Compute Engine related pages of the Developers Console. The gcloud command line continues to function as normal. We will provide another status update by 16:00 PDT.

We are still investigating the issue with the Developers Console where some users are seeing errors when browsing the Compute Engine related pages of the Developers Console. The gcloud command line continues to function as normal. We will provide another status update by 15:30 PDT.

We are still investigating the issue with the Developer's Console and the Compute Engine related pages. We will provide another status update by 2015-04-08 14:50 US/Pacific.

We are currently experiencing an issue with the Developers Console and some users are seeing errors when accessing Compute Engine specific pages. For everyone who is affected, we apologize for any inconvenience you may be experiencing.