Service Health
-
-
-
-
Incident affecting Google Compute Engine
Issue creating and deleting Compute Engine instances
Incident began at 2020-07-14 22:34 and ended at 2020-07-15 02:14 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 15 Jul 2020 | 18:00 PDT | Initially, we believed that this incident affected all instance create and delete operations, which was inaccurate. Approximately 2% of requests globally were failing and the issue was transient, allowing retried operations to succeed. All existing instances continued to operate as expected, with only create or delete operations being affected. Upon evaluation of Google’s incident scope and severity, this incident’s severity has been adjusted from Outage to Disruption. |
| | 15 Jul 2020 | 09:14 PDT | The issue with creating and deleting Google Compute Engine instances has been resolved for all affected projects as of Wednesday, 2020-07-15 08:56 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 15 Jul 2020 | 09:10 PDT | Description: We are experiencing an issue creating and deleting Google Compute Engine instances, starting at Wednesday, 2020-07-15 05:07 US/Pacific. This is also affecting other products which rely on Compute Engine including Cloud Build, which is in turn used by services including App Engine and Cloud Functions. Other products affected are Cloud Data Fusion, Cloud Dataproc, Cloud Dataflow, and Google Kubernetes Engine. As well, instances created may experience DNS propagation delays. Current data indicates that approximately 2% of requests globally are affected by this issue. Mitigation work has been completed by our engineering team and we are monitoring recovery. Full resolution is expected to complete by Wednesday, 2020-07-15 09:05 US/Pacific. We will provide an update by Wednesday, 2020-07-15 09:20 US/Pacific with current details. Diagnosis: Attempts to create a new instance or to delete an instance fail with an error. Workaround: Retrying the instance operation may succeed. |
| | 15 Jul 2020 | 08:50 PDT | Description: We are experiencing an issue creating and deleting Google Compute Engine instances, starting at Wednesday, 2020-07-15 05:07 US/Pacific. This is also affecting other products which rely on Compute Engine including Cloud Build, which is in turn used by services including App Engine and Cloud Functions. Other products affected are Cloud Data Fusion, Cloud Dataproc, Cloud Dataflow, and Google Kubernetes Engine. Current data indicates that approximately 2% of requests globally are affected by this issue. Mitigation work has been completed by our engineering team and we are monitoring recovery. Full resolution is expected to complete by Wednesday, 2020-07-15 09:05 US/Pacific. We will provide an update by Wednesday, 2020-07-15 09:15 US/Pacific with current details. Diagnosis: Attempts to create a new instance or to delete an instance fail with an error. Workaround: Retrying the instance operation may succeed. |
| | 15 Jul 2020 | 08:11 PDT | Description: We are experiencing an issue creating and deleting Google Compute Engine instances, starting at Wednesday, 2020-07-15 05:07 US/Pacific. This is also affecting other products which rely on Compute Engine including Cloud Build, which is in turn used by services including App Engine and Cloud Functions. Other products affected are Cloud Data Fusion, Cloud Dataproc, and Google Kubernetes Engine. Mitigation work is still underway by our engineering team. Current data indicates that approximately 2% of requests globally are affected by this issue. We do not have an ETA for mitigation at this point. We will provide an update by Wednesday, 2020-07-15 09:00 US/Pacific with current details. Diagnosis: Attempts to create a new instance or to delete an instance fail with an error. Workaround: Retrying the instance operation may succeed. |
| | 15 Jul 2020 | 07:48 PDT | Description: We are experiencing an issue creating and deleting Google Compute Engine instances, starting at Wednesday, 2020-07-15 05:07 US/Pacific. This is also affecting other products which rely on Compute Engine including Cloud Build, which is in turn used by services including App Engine and Cloud Functions. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide an update by Wednesday, 2020-07-15 08:20 US/Pacific with current details. Diagnosis: Attempts to create a new instance or to delete an instance fail with an error. Workaround: Retrying the instance operation may succeed. |
| | 15 Jul 2020 | 06:59 PDT | Description: We are experiencing an issue creating and deleting Google Compute Engine instances, starting at Wednesday, 2020-07-15 05:07 US/Pacific. This is also affecting other products which rely on Compute Engine including Cloud Build, which is in turn used by services including App Engine and Cloud Functions. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2020-07-15 08:00 US/Pacific with current details. Diagnosis: Attempts to create a new instance or to delete an instance fail with an error. Workaround: None at this time. |
| | 15 Jul 2020 | 06:11 PDT | Description: We are experiencing an issue creating new instances with Google Compute Engine, starting at Wednesday, 2020-07-15 05:07 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2020-07-15 07:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Attempts to create a new instance fail with an error. Workaround: None at this time. |
- All times are US/Pacific