Service Health

The issue with Google Compute Engine operations in us-central1-a has been resolved for all affected projects as of Wednesday, 2020-01-29 16:16 US/Pacific.

We thank you for your patience while we’ve worked on resolving the issue.

Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations.

Running instances were not impacted. Other zones in us-central1 were not impacted. New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. Mitigation efforts continue to accelerate the backlog processing.

Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore.

We will provide an update by Wednesday, 2020-01-29 16:30 US/Pacific with current details.

Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”.

Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations.

Running instances were not impacted. Other zones in us-central1 were not impacted.

Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore.

We will provide an update by Wednesday, 2020-01-29 14:30 US/Pacific with current details.

Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”.

Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations.

Stop and delete operations between Wednesday, 2020-01-29 04:00 US/Pacific and 09:00 US/Pacific may have failed or appear stuck. Stop and delete operations created after 09:00 US/Pacific should complete as expected, even if they had previously failed or are stuck.

Mitigation is underway and we are currently working through the backlog of operations.

Currently running instances are not impacted. Other zones in us-central1 are not impacted.

Products which depend on GCE may also be unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore

We will provide an update by Wednesday, 2020-01-29 11:00 US/Pacific with current details.

Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”.

Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation.

Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Impact began Wednesday, 2020-01-29 06:02 US/Pacific. Mitigation is underway and we are currently working through the backlog operations.

Currently running instances are not impacted.

We will provide an update by Wednesday, 2020-01-29 10:00 US/Pacific with current details.

Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”.

Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation.

Workaround: None at this time.

Description: We are experiencing an issue with Google Compute Engine affecting mainly us-central1-a, although other zones in the region may see the issue as well, though at a much lower rate. Instance STOP and DELETE requests may fail or complete with an error, beginning Wednesday, 2020-01-29 06:02 US/Pacific.

Symptoms: Instance STOP and DELETE requests may fail or complete with an error.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2020-01-29 08:45 US/Pacific with current details.

Diagnosis: Instance STOP and DELETE requests may fail or complete with an error.

Workaround: None at this time.