Service Health

SUMMARY:

On Saturday 31 October 2015, Google Compute Engine (GCE) management operations experienced high latency for a duration of 181 minutes. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT:

On Saturday 31 October 2015 from 18:04 to 21:05 PDT, all Google Compute Engine management operations were slow or timed out in the Google Developers Console, the gcloud tool or the Google Compute Engine API.

ROOT CAUSE:

An issue in the handling of Google Compute Engine management operations caused requests to not complete in a timely manner, due to older operations retrying excessively and preventing newer operations from succeeding.

Once discovered, remediation steps were taken by Google Engineers to reduce the number of retrying operations, enabling recovery from the operation backlog. The incident was resolved at 21:05 PDT when all backlogged operations were processed by the Google Compute Engine management backend and latency and error rates returned to typical values.

REMEDIATION AND PREVENTION:

To detect similar situations in the future, the GCE Engineering team is implementing additional automated monitoring to detect high numbers of queued management operations and limiting the number of operation retries. Google Engineers are also enabling more robust operation handling and load splitting to better isolate system disruptions.

The issue with Google Compute Engine Instance operation high latency should have been resolved for all affected users as of 21:05 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are still investigating the issue with Google Compute Engine Instance operation high latency. We will provide another status update by 22:00 US/Pacific with current details.

We are experiencing an issue with Google Compute Engine Instance operation high latency beginning at Saturday, 2015-10-31 18:04 US/Pacific.

Current data indicates that only users who are attempting to run instance management operations are affected by this issue.

For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 21:00 US/Pacific with current details.

We are investigating reports of an issue with Google Compute Engine Instance operations. We will provide more information by 2015-10-31 19:30 US/Pacific.