Google Cloud Status Dashboard
This page provides status information on the services that are part of Google Cloud Platform. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit cloud.google.com.
Google Compute Engine Incident #20001
The issue with instance DELETE and STOP operations in Google Compute Engine has been mitigated.
Incident began at 2020-01-28 12:18 and ended at 2020-01-29 16:26 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| Jan 29, 2020 | 16:26 | The issue with Google Compute Engine operations in us-central1-a has been resolved for all affected projects as of Wednesday, 2020-01-29 16:16 US/Pacific. We thank you for your patience while we’ve worked on resolving the issue. |
|
The issue with Google Compute Engine operations in us-central1-a has been resolved for all affected projects as of Wednesday, 2020-01-29 16:16 US/Pacific. We thank you for your patience while we’ve worked on resolving the issue. |
|||
| Jan 29, 2020 | 14:09 | Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. Mitigation efforts continue to accelerate the backlog processing. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 16:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|
Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. Mitigation efforts continue to accelerate the backlog processing. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 16:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|||
| Jan 29, 2020 | 10:52 | Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 14:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|
Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 14:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|||
| Jan 29, 2020 | 09:56 | Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Stop and delete operations between Wednesday, 2020-01-29 04:00 US/Pacific and 09:00 US/Pacific may have failed or appear stuck. Stop and delete operations created after 09:00 US/Pacific should complete as expected, even if they had previously failed or are stuck. Mitigation is underway and we are currently working through the backlog of operations. Currently running instances are not impacted. Other zones in us-central1 are not impacted. Products which depend on GCE may also be unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore We will provide an update by Wednesday, 2020-01-29 11:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|
Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Stop and delete operations between Wednesday, 2020-01-29 04:00 US/Pacific and 09:00 US/Pacific may have failed or appear stuck. Stop and delete operations created after 09:00 US/Pacific should complete as expected, even if they had previously failed or are stuck. Mitigation is underway and we are currently working through the backlog of operations. Currently running instances are not impacted. Other zones in us-central1 are not impacted. Products which depend on GCE may also be unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore We will provide an update by Wednesday, 2020-01-29 11:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. |
|||
| Jan 29, 2020 | 08:44 | Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Impact began Wednesday, 2020-01-29 06:02 US/Pacific. Mitigation is underway and we are currently working through the backlog operations. Currently running instances are not impacted. We will provide an update by Wednesday, 2020-01-29 10:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: None at this time. |
|
Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Impact began Wednesday, 2020-01-29 06:02 US/Pacific. Mitigation is underway and we are currently working through the backlog operations. Currently running instances are not impacted. We will provide an update by Wednesday, 2020-01-29 10:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: None at this time. |
|||
| Jan 29, 2020 | 07:35 | Description: We are experiencing an issue with Google Compute Engine affecting mainly us-central1-a, although other zones in the region may see the issue as well, though at a much lower rate. Instance STOP and DELETE requests may fail or complete with an error, beginning Wednesday, 2020-01-29 06:02 US/Pacific. Symptoms: Instance STOP and DELETE requests may fail or complete with an error. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2020-01-29 08:45 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error. Workaround: None at this time. |
|
Description: We are experiencing an issue with Google Compute Engine affecting mainly us-central1-a, although other zones in the region may see the issue as well, though at a much lower rate. Instance STOP and DELETE requests may fail or complete with an error, beginning Wednesday, 2020-01-29 06:02 US/Pacific. Symptoms: Instance STOP and DELETE requests may fail or complete with an error. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2020-01-29 08:45 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error. Workaround: None at this time. |
|||