Service Health

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 31 August 2021 19:35

Incident End: 2 September 2021 20:55

Duration: 2 days, 1 hours, 20 minutes

Affected Services and Features:

• Google Compute Engine - Local SSD

Regions/Zones:

• us-central1-{a|b|f}
• europe-west1-b
• us-west1-a
• asia-east1-b

Description:

Google Compute Engine (GCE) experienced local solid-state drive (SSD) unavailability for new instances in select zones intermittently for a duration of 2 days, 1 hours, and 20 minutes. From preliminary analysis, the root cause is due to latent issues within the GCE capacity management system.

Customer Impact:

• Unable to add Local SSDs to existing/new instances.

Additional details:

• The asia-southeast1-b zone was not impacted despite previous communication that suggested it was.
• Existing instances, or instances that rebooted, were not impacted.
• Low priority maintenance events on host machines were paused to prevent further impact to customer instances.

The issue with VM maintenance on Google Compute Engine has been resolved for all affected users as of Friday, 2021-09-03 08:31 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Mitigation work is still underway by our engineering team. The engineering team has confirmed that pausing the VM maintenance has mitigated current customer impact.

Zones recovered: us-west1, asia-east1, and asia-southeast1 Zones recovering: us-central1, europe-west1

Suspected Root cause: An internal maintenance operation incorrectly removed too many machines from the service.

We will provide more information by Friday, 2021-09-03 13:30 US/Pacific.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Mitigation work is currently under progress. Engineering has confirmed that pausing the VM maintenance has mitigated current customer impact.

Customers may observe loss of local SSD on VM termination till the full resolution is deployed.

We will provide more information by Friday, 2021-09-03 13:30 US/Pacific.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Mitigation work is currently under progress. Engineering team continues to pause the regular VM maintenance operations for SSD machines to mitigate the customer impact.

We do not have an ETA for completion of mitigation activities at this point.

We will provide more information by Thursday, 2021-09-02 13:30 US/Pacific.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Engineering team has implemented a pause on regular VM maintenance operations for SSD machines to mitigate the customer impact.

We do not have an ETA for completion of mitigation activities at this point.

We will provide more information by Thursday, 2021-09-02 12:00 US/Pacific.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Engineering team has started mitigation efforts to reduce customer impact.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2021-09-02 11:00 US/Pacific.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.

Summary: us-central1, europe-west1, us-west1, asia-east1, asia-southeast1: Issue with VM maintenance on Google Compute Engine.

Description: Some customers are experiencing an issue performing VM maintenance activities on Google Compute Engine.

Our engineering team is currently investigating the issue.

We will provide an update by Thursday, 2021-09-02 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may see on VM termination/maintenance, replacement VMs cannot acquire Local SSD in order to initialize and therefore VMs will not be able to restart.

Workaround: No workaround available at this time.