Service Health
-
-
-
-
Incident affecting Google Cloud Composer
Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions.
Incident began at 2023-08-01 23:00 and ended at 2023-08-02 03:45 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Melbourne (australia-southeast2)Belgium (europe-west1)London (europe-west2)Paris (europe-west9)Iowa (us-central1)Salt Lake City (us-west3)
| Date | Time | Description | |
|---|---|---|---|
| | 2 Aug 2023 | 11:21 PDT | Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . (All Times US/Pacific) Incident Start: 1 August 2023 at 23:00 Incident End: 2 August 2023 at 03:45 Duration: 4 hours, 45 minutes Affected Services and Features: Cloud Composer Regions/Zones:
Description: Cloud Composer 2 environments experienced creation, update, and snapshots operation failures for a duration of 4 hours and 45 minutes in multiple regions. From preliminary analysis, the root cause was the invalid proxy configuration, which blocked outgoing traffic to Google Kubernetes Engine (GKE). Customer Impact: Customers experienced failures when attempting the following actions:
DAGs executions were not impacted by the outage as long as configuration changes were not being introduced. |
| | 2 Aug 2023 | 04:50 PDT | The issue with Google Cloud Composer has been resolved for all affected projects as of Wednesday, 2023-08-02 04:49 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 2 Aug 2023 | 04:43 PDT | Summary: Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions. Description: Mitigation work is currently underway and we believe the issue is mostly resolved at this time. We're performing checks to confirm and we will provide an update by Wednesday, 2023-08-02 06:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update and snapshot operation failing. Workaround: Customers are recommended to not perform any update operations until this incident is over and refrain from creating Composer environments in the impacted regions. DAG executions in existing environments should work fine assuming that the customer doesn’t try to introduce changes related to Airflow components reconfiguration. |
| | 2 Aug 2023 | 03:27 PDT | Summary: Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions:
Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-08-02 05:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update and snapshot operation failing. Workaround: Customers are recommended to not perform any update operations until this incident is over and refrain from creating Composer environments in the impacted regions. DAG executions in existing environments should work fine assuming that the customer doesn’t try to introduce changes related to Airflow components reconfiguration. |
| | 2 Aug 2023 | 02:13 PDT | Summary: Cloud Composer 2 customers experiencing create and update operation failures in multiple regions:
Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-08-02 04:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update operation failing Workaround: None at this time. |
| | 2 Aug 2023 | 01:52 PDT | Summary: Cloud Composer 2 customers experiencing create and update operation failures in multiple regions Description: We are experiencing an issue with Google Cloud Composer 2 beginning on Wednesday, 2023-08-01 22:30 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2023-08-02 03:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update operation failing Workaround: None at this time. |
- All times are US/Pacific