Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 13 July 2023 at 07:55

Incident End: 21 July 2023 at 02:29

Duration: 7 days, 18 hours, 34 minutes

Affected Services and Features:

Cloud Data Fusion

Regions/Zones: asia-northeast2, me-west1, us-east1, us-east4, us-east5, us-central1, us-south1, us-west2

Description:

Cloud Data Fusion experienced an issue where updating an instance created before 14 July 2023 would fail.

Our preliminary analysis indicates that the root cause is a recent change made to access controls on Google Kubernetes Engine (GKE) clusters in order to improve security. Due to this change, the Cloud Data Fusion instances using older service accounts did not have the appropriate permissions to apply instance updates.

Google Engineers mitigated the issue by updating affected instances to use the correct service account.

Customer Impact:

Customers from the impacted regions would have observed failures during instance updates for instances that were created before 14 July 2023. However, there was no known instance usage impact for the customers who did not update their instances.

The issue with Cloud Data Fusion has been resolved for all affected users as of Friday, 2023-07-21 02:55 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team. Currently our engineers have mitigated us-east5, us-south1, asia-northeast2, us-west2, us-east4 and me-west1,us-central1

We are actively working on mitigating us-east1

We will provide more information by Friday, 2023-07-21 04:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team. Currently our engineers have mitigated us-east5, us-south1, asia-northeast2, us-west2, us-east4 and me-west1. We are actively working on mitigating us-east1 and us-central1.

We will provide more information by Friday, 2023-07-21 03:00 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team. Currently our engineers have mitigated us-east5, us-south1, asia-northeast2, us-west2, and me-west1 and are actively working on mitigating other regions.

We will provide more information by Friday, 2023-07-21 01:00 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team. Currently our engineers have mitigated us-east5 and us-south1 and are actively working on mitigating other regions.

We will provide more information by Thursday, 2023-07-20 21:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team.

We will provide more information by Thursday, 2023-07-20 17:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is still underway by our engineering team.

We will provide more information by Thursday, 2023-07-20 15:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2023-07-20 13:30 US/Pacific.

We will provide more information by Thursday, 2023-07-20 15:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2023-07-20 13:30 US/Pacific.

We will provide more information by Thursday, 2023-07-20 14:30 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2023-07-20 13:30 US/Pacific.

We will provide more information by Thursday, 2023-07-20 14:00 US/Pacific.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: We are experiencing an issue with Cloud Data Fusion, the instance usage will not be impacted.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2023-07-20 03:30 US/Pacific with current details.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions. However, the instance usage will not be impacted.

Workaround: None at this time.

Summary: Cloud Data Fusion instance update failure.

Description: We are experiencing an issue with Cloud Data Fusion.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-07-19 23:20 US/Pacific with current details.

Diagnosis: Cloud Data Fusion instances created before July 14, 2023 failing during instance upgrade in a few regions.

Workaround: None at this time.