SUMMARY:

On Tuesday 23 February 2016, for a duration of 10 hours and 6 minutes, 7.8% of Google Compute Engine projects had reduced quotas. We know that the ability to scale is vital to our customers, and apologize for preventing you from using the resources you need.

DETAILED DESCRIPTION OF IMPACT:

On Tuesday 23 February 2016 from 06:06 to 16:12 PST, 7.8% of Google Compute Engine projects had quotas reduced. This impacted all quotas, including number of cores, IP addresses and disk size. If reduced quota was applied to your project and your usage reached this reduced quota you would have been unable to create new resources during this incident. Any such attempt would have resulted in a QUOTA_EXCEEDED error code with message "Quota 'XX_XX' exceeded. Limit: N". Any resources that were already created were unaffected by this issue.

ROOT CAUSE:

In order to maximize ease of use for Google Compute Engine customers, in some cases we automatically raise resource quotas. We then provide exclusions to ensure that no quotas previously raised are reduced. We occasionally tune the algorithm to determine which quotas can be safely raised. This incident occurred when one such change was made but a bug in the aforementioned exclusion process allowed some projects to have their quotas reduced.

REMEDIATION AND PREVENTION:

As soon as Google engineers identified the cause of the issue the initiating change was rolled back and quota changes were reverted. To provide faster resolution to quota related issues in the future we are creating new automated alerting and operational documentation. To prevent a recurrence of this specific issue, we have fixed the bug in the exclusion process. To prevent similar future issues, we are also creating a dry-run testing phase to verify the impact quota system changes will have.

The issue with quotas being reset to default values should have been resolved for all affected customers as of 16:30 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

The issue with quotas being reset to default values should have been resolved for the majority of projects and we expect a full resolution in the near future. We will provide another status update by 16:30 US/Pacific with current details.

We are continuing to investigate the issue with quotas being reset to default values for some customers. We'll provide a new update at 15:30 US / Pacific time.

We are continuing to investigate the issue with quotas being reset to default values for some customers. We'll provide a new update at 14:30 US / Pacific time.

We are continuing to investigate the issue with quotas being reset to default values for some customers. We'll provide a new update at 13:30 Pacific time

We continue to investigate the problem with quotas being reset to default values for some customers. We'll provide a new update at 12:30 PT.

We are still investigating the problem of some projects quotas being reverted back to Default values and we'll provide a new update in at 11:30 PST time.

We are investigating a problem with our Quota System where Quotas were reset to default values for some Customers.