Service Health
Incident affecting Google Cloud Composer
Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected.
Incident began at 2023-08-25 06:06 and ended at 2023-09-04 07:30 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Paris (europe-west9)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
Date | Time | Description | |
---|---|---|---|
| 4 Sep 2023 | 07:30 PDT | The issue with Google Cloud Composer has been resolved for all affected users as of Monday, 2023-09-04 07:13 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| 1 Sep 2023 | 06:52 PDT | Summary: Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected. Description: Our engineering team has deployed the fix that is currently in propagation. Release note was published for the customers. Please refer to it for further details: https://cloud.google.com/composer/docs/release-notes#September_01_2023 We do not have an ETA for mitigation at this point but we will provide an update by Monday, 2023-09-04 17:00 US/Pacific with current details. Diagnosis: Functionality of blocking and allowing access to Cloud Composer’s Airflow UI and Airflow REST API via VPC Service Controls perimeter and access levels doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). Any users who meet the conditions presented below have access to Airflow UI and Airflow REST API even when you block their access through VPC Service Controls settings. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 28 Aug 2023 | 08:08 PDT | Summary: Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team has identified a fix and plans to deploy it in the next few days. Once the fix is deployed, VPC Service Controls perimeter settings (like ingress rules) and access level settings will start to be enforced when accessing Airflow UI and Airflow REST API, while previously they might have been ignored. As a result, some Composer users might not be able to access Airflow UI or Airflow REST API. In such a case, please update the perimeter or access level configurations accordingly to match your needs. We will provide an update by Friday, 2023-09-01 10:00 US/Pacific with current details. Diagnosis: Functionality of blocking and allowing access to Cloud Composer’s Airflow UI and Airflow REST API via VPC Service Controls perimeter and access levels doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). Any users who meet the conditions presented below have access to Airflow UI and Airflow REST API even when you block their access through VPC Service Controls settings. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 28 Aug 2023 | 00:44 PDT | Summary: Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team has identified a fix and plans to deploy it in the next few days. We will provide an update by Friday, 2023-09-01 10:00 US/Pacific with current details. Diagnosis: Functionality of blocking and allowing access to Cloud Composer’s Airflow UI and Airflow REST API via VPC Service Controls perimeter and access levels doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). Any users who meet the conditions presented below have access to Airflow UI and Airflow REST API even when you block their access through VPC Service Controls settings. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 25 Aug 2023 | 08:59 PDT | Summary: Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-08-28 01:00 US/Pacific with current details. Diagnosis: Functionality of blocking and allowing access to Cloud Composer’s Airflow UI and Airflow REST API via VPC Service Controls perimeter and access levels doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). Any users who meet the conditions presented below have access to Airflow UI and Airflow REST API even when you block their access through VPC Service Controls settings. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 25 Aug 2023 | 08:22 PDT | Summary: Functionality of controlling access to Cloud Composer’s Airflow UI and Airflow REST API (Composer 2 version) via VPC Service Controls mechanism might not work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2023-08-25 11:00 US/Pacific with current details. Diagnosis: Functionality of blocking and allowing access to Cloud Composer’s Airflow UI and Airflow REST API via VPC Service Controls perimeter and access levels doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). Any users who meet the conditions presented below have access to Airflow UI and Airflow REST API even when you block their access through VPC Service Controls settings. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 25 Aug 2023 | 08:20 PDT | Summary: Functionality of explicitly defining the list of users allowed to access Cloud Composer’s Airflow UI (Composer 2 version) via VPC Service Controls mechanism doesn’t work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2023-08-25 11:00 US/Pacific with current details. Diagnosis: Functionality of explicitly defining the list of users allowed to access Cloud Composer’s Airflow UI via ingress policies defined within in VPC Service Controls’s access level doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). As a result of it, regardless of what is configured in the access policy, any users who meet the conditions presented below have access to Airflow UI. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
| 25 Aug 2023 | 06:52 PDT | Summary: Functionality of explicitly defining the list of users allowed to access Cloud Composer’s Airflow UI (Composer 2 version) via VPC Service Controls mechanism doesn’t work as expected. Description: We are experiencing an issue with Google Cloud Composer beginning on Friday, 2023-08-25 05:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2023-08-25 09:00 US/Pacific with current details. Diagnosis: Functionality of explicitly defining the list of users allowed to access Cloud Composer’s Airflow UI via ingress policies defined within in VPC Service Controls’s access level doesn’t work as expected (applicable to Composer version 2 users, it is not affecting Composer version 1 users). As a result of it, regardless of what is configured in the access policy, any users who meet the conditions presented below have access to Airflow UI. Important: Regardless of this issue, to have access to Airflow UI, a user:
Workaround: Rely only on Cloud IAM permissions and Airflow RBAC controls (https://cloud.google.com/composer/docs/airflow-rbac) to give or decline user access to Airflow UI. |
- All times are US/Pacific