Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Compute Engine

Google Compute Engine experiencing issues in us-central1 region

Incident began at 2023-08-08 20:00 and ended at 2023-08-09 01:56 (all times are US/Pacific).

Previously affected location(s)

Iowa (us-central1)

Date Time Description
9 Aug 2023 17:23 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 08 August 2023 at 20:00

Incident End: 09 August 2023 at 01:56

Duration: 5 hours, 56 minutes

Affected Services and Features:

Google Compute Engine (GCE)

Regions/Zones: us-central1

Description:

GCE experienced elevated timeouts and error rates on a subset of API requests for a duration of 5 hours and 56 minutes.

Affected API requests include:

  • compute.images.insert, compute.images.delete API requests to disks in us-central1-a and some regional disks in us-central1
  • compute.regionDisks.createSnapshot API requests in us-central1
  • compute.disks.createSnapshot in us-central1-a

From the preliminary analysis, the root cause of the issue are failures in Persistent Disk devices snapshots [1] that were traced to one device which inturn affected our capability to serve snapshots in this cell .

Google Engineers mitigated the issue by restarting all the tasks within the impacted cell.

[1] - https://cloud.google.com/compute/docs/disks/snapshots

Customer Impact:

Approximately 25% of customers of Google Compute Engine, would have faced errors with the API requests mentioned in the description section above. Retrying the API requests may result in completion of the requests.
9 Aug 2023 02:20 PDT

The issue with Google Compute Engine has been resolved for all affected users as of Wednesday, 2023-08-09 01:55 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
9 Aug 2023 02:17 PDT

Summary: Google Compute Engine experiencing issues in us-central1 region

Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2023-08-09 03:00 US/Pacific.

We will provide more information by Wednesday, 2023-08-09 03:00 US/Pacific.

Diagnosis: Customers impacted by this issue may see operations for the following API methods failing: compute.disks.createSnapshot, compute.images.delete, compute.images.insert and compute.regionDisks.createSnapshot

Workaround: None at this time.
9 Aug 2023 02:05 PDT

Summary: Google Compute Engine experiencing issues in us-central1 region

Description: We are experiencing an issue with Google Compute Engine beginning on Tuesday, 2023-08-08 20:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-08-09 03:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers impacted by this issue may see operations for the following API methods failing: compute.disks.createSnapshot, compute.images.delete, compute.images.insert and compute.regionDisks.createSnapshot

Workaround: None at this time.