Service Health
Incident affecting Google Cloud Composer
We are mitigating an issue with creating and deleting Cloud Composer environments.
Incident began at 2019-07-25 05:13 and ended at 2019-07-25 19:59 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 1 Aug 2019 | 11:44 PDT | After further investigation, we have determined that the impact was minimal and affected a subset of users. We have conducted an internal investigation of this issue and made appropriate improvements to our systems to help prevent or minimize a future recurrence. |
| 25 Jul 2019 | 18:31 PDT | The issue with Cloud Composer environment creation/deletion errors has been resolved for all affected projects as of Thursday, 2019-07-25 18:31 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation. |
| 25 Jul 2019 | 16:51 PDT | The issue with Cloud Composer environment creation/deletion errors should be resolved for the majority of projects. Currently, approximately 5% of the requests are still failing. Our engineers are still working to mitigate the issue and we expect a full resolution in the near future. We will provide another status update by Thursday, 2019-07-25 18:00 US/Pacific with current details. |
| 25 Jul 2019 | 16:42 PDT | The issue with Cloud Composer environment creation/deletion errors should be resolved for the majority of projects. Currently, approximately 5% of the requests are still failing. Our engineers are still working to mitigate the issue and we expect a full resolution in the near future. We will provide another status update by Thursday, 2019-07-25 18:00 US/Pacific with current details. |
| 25 Jul 2019 | 15:39 PDT | The rate of errors is decreasing. We will provide another status update by Thursday, 2019-07-25 16:45 US/Pacific with current details. |
| 25 Jul 2019 | 14:42 PDT | We are still seeing errors. Our Engineers have implemented a mitigation and are now monitoring the situation before gradually rolling it out to a wider audience. We will provide another status update by Thursday, 2019-07-25 15:45 US/Pacific with current details. |
| 25 Jul 2019 | 13:43 PDT | We are still seeing errors. Our Engineers are attempting several mitigation strategies in parallel and are beginning to see positive results. We will provide another status update by Thursday, 2019-07-25 14:45 US/Pacific with current details. |
| 25 Jul 2019 | 12:29 PDT | Mitigation work is currently still underway by our Engineering Team. We will provide another status update by Thursday, 2019-07-25 13:45 US/Pacific with current details. |
| 25 Jul 2019 | 10:34 PDT | We are investigating errors with creating and deleting Cloud Composer environments. Our engineering team has identified a root cause, and mitigation work is currently underway. Efforts to accelerate recovery using the previous mitigation were not effective and the issue has been escalated internally, we are continuing to work on a new mitigation. We will provide another status update by Thursday, 2019-07-25 12:30 US/Pacific with current details. |
| 25 Jul 2019 | 09:14 PDT | We are investigating errors with creating and deleting Cloud Composer environments. Our engineering team has identified a root cause, and mitigation work is currently underway. Mitigation appears effective, and we are working to accelerate recovery. We will provide another status update by Thursday, 2019-07-25 10:30 US/Pacific with current details. |
| 25 Jul 2019 | 08:36 PDT | We are investigating errors with creating and deleting Cloud Composer environments. Our Engineering Team believes they have identified the root cause and mitigation work is underway. We will provide another status update by Thursday, 2019-07-25 09:30 US/Pacific with current details. |
- All times are US/Pacific