Service Health

This is a preliminary Incident Report. We are preparing a Full Incident Report with comprehensive details which will be posted on a later date.

We apologize for the impact this service outage may have caused. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

On Tuesday, 25 April at 17:20 US/Pacific a water leak in one of our data centers in Paris led to a fire in a battery room. Subsequently, Google experienced an infrastructure failure that affected our europe-west-9 Cloud region, impacting multiple Google Cloud Services.

The water leak initially impacted a portion of europe-west9-a; however, the subsequent fire required all of europe-west9-a and a portion of europe-west9-c to be temporarily powered down. Many regional services were affected while europe-west9-c was partially unavailable. These regional services were restored once europe-west9-c and part of europe-west9-a returned to service.

Over the course of this incident, some services were completely unavailable for a short period, while others were only partially available. The following are some of the impacts of the outage:

• Global Service Impact - Customers may have experienced issues with Google Cloud Console and a subset of Compute APIs which are designed to serve all zones worldwide starting around 25 April at 23:15 US/Pacific. In addition, customers who use the Global Scope for Compute internal DNS names were unable to create new instances if they had instances running in europe-west9 at the time of the outage. Services were restored by 26 April at 03:45 US/Pacific.
• Regional Service Impact - Regional services including GCS and GCLB were unavailable starting at 25 April 23:05 US/Pacific and were recovered by 26 April at 22:49 US/Pacific. These services were unavailable due to the full shutdown of europe-west9-a and the partial shutdown of europe-west9-c. Additionally, many zonal services in the region, such as Compute Engine and Persistent Disk, were partially or fully unavailable due to impacts to the regional control plane or their underlying zones.
• Zonal Service Impact - (partially continuing) - Zonal services in europe-west9-a were unavailable from 25 April at 18:34 US/Pacific until 26 April at 17:45 US/Pacific when they were partially recovered. Some zonal control planes had some functionality impacted until 4 May at 15:30 US/Pacific. Impacted instances that have recovered and new instances are expected to function normally. Some instances located in the impacted portions of the datacenter remain unavailable. Google has contacted customers with impacted resources directly.

Google teams are currently working directly with customers who were impacted in the europe-west9-a zone.

Google will continue to provide updates to customers with open support cases and will work with them until their services are restored. The Full Incident Report will detail the changes we are making to eliminate the global, regional, and zonal service impact, as well as the overall improvements we will be making based on the RCA. No further updates will be provided on the Cloud Status Dashboard for this incident until we publish the Incident Report.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-10 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-09 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-09 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Monday, 2023-05-08 13:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Monday, 2023-05-08 02:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Thursday, 2023-05-04 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Thursday, 2023-05-04 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-03 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-03 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The list of current impacted products has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Tuesday, 2023-05-02 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Tuesday, 2023-05-02 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Tuesday, 2023-05-02 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Tuesday, 2023-05-02 04:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Monday, 2023-05-01 11:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Sunday, 2023-04-30 23:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Friday, 2023-04-28 19:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Friday, 2023-04-28 07:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

We will provide an update by Thursday, 2023-04-27 19:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone.

Workaround: Customers can failover to other zones in europe-west9 or to other regions.

Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expected to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted.

The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM).

The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL

We will provide an update by Thursday, 2023-04-27 11:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.

The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE)

The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: The issue with Cloud Pub/Sub has been resolved for all affected users as of Wednesday, 2023-04-26 16:07 S/Pacific. For more information please see here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Thursday, 2023-04-27 07:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.

The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE)

The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Thursday, 2023-04-27 03:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.

The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE)

The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Thursday, 2023-04-27 03:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Description: Water intrusion in a datacenter in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Thursday, 2023-04-27 00:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Wednesday, 2023-04-26 20:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

Cloud Console: Experienced global outages which have been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT.

GCE Control Plane: Experienced global outages, which have been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT and 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). Secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal

Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4

BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG

We will provide an update by Wednesday, 2023-04-26 16:00 US/Pacific, or upon any significant development.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions.

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 12:00 US/Pacific, or upon any significant development.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 12:00 US/Pacific, or upon any significant development.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

For management tasks Cloud Console should be operational again for operations outside the affected region (europe-west9).

We will provide an update by Wednesday, 2023-04-26 07:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

For management tasks Cloud Console should be operational again for operations outside the affected region (europe-west9)

We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Customer should use gcloud commands instead of Cloud Console for management tasks

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 05:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Workaround: Customers can failover to zones in other regions

Customer should use gcloud commands instead of Cloud Console for management tasks

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 04:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region.

Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page

Workaround: Customers can failover to zones in other regions

Customer should use gcloud commands instead of Cloud Console for management tasks

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 03:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page

Workaround: Customers can failover to zones in other regions Customer should use gcloud commands instead of Cloud Console for management tasks

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 03:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted.

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 02:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region

Workaround: Customers can failover to zones in other regions

Summary: Multiple Google Cloud services in the europe-west9 region are impacted

Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.

We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region

Workaround: Customers can failover to zones in other regions

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones if they are impacted.

We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted.

We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones within europe-west9

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted.

We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other available zones

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted.

We will provide an update by Tuesday, 2023-04-25 22:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones within europe-west9

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-04-25 21:00 US/Pacific with current details.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones within europe-west9

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-04-25 20:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones within europe-west9

Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-04-25 19:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a

Workaround: Customers can fail over to other zones within europe-west9