Service Health
Incident affecting Google Compute Engine, Google Kubernetes Engine, Google Cloud Bigtable, Persistent Disk, Google Cloud SQL, Cloud Filestore
Multiple Google Cloud services in the europe-west9-a zone are impacted
Incident began at 2023-04-25 19:00 and ended at 2023-05-10 15:38 (all times are US/Pacific).
Previously affected location(s)
Multi-region: europeParis (europe-west9)Global
Date | Time | Description | |
---|---|---|---|
| 10 May 2023 | 15:38 PDT | This is a preliminary Incident Report. We are preparing a Full Incident Report with comprehensive details which will be posted on a later date. We apologize for the impact this service outage may have caused. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . On Tuesday, 25 April at 17:20 US/Pacific a water leak in one of our data centers in Paris led to a fire in a battery room. Subsequently, Google experienced an infrastructure failure that affected our europe-west-9 Cloud region, impacting multiple Google Cloud Services. The water leak initially impacted a portion of europe-west9-a; however, the subsequent fire required all of europe-west9-a and a portion of europe-west9-c to be temporarily powered down. Many regional services were affected while europe-west9-c was partially unavailable. These regional services were restored once europe-west9-c and part of europe-west9-a returned to service. Over the course of this incident, some services were completely unavailable for a short period, while others were only partially available. The following are some of the impacts of the outage:
Google teams are currently working directly with customers who were impacted in the europe-west9-a zone. Google will continue to provide updates to customers with open support cases and will work with them until their services are restored. The Full Incident Report will detail the changes we are making to eliminate the global, regional, and zonal service impact, as well as the overall improvements we will be making based on the RCA. No further updates will be provided on the Cloud Status Dashboard for this incident until we publish the Incident Report. |
| 9 May 2023 | 15:54 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-10 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 9 May 2023 | 03:15 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-09 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 8 May 2023 | 12:11 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-09 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 8 May 2023 | 01:17 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Monday, 2023-05-08 13:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 5 May 2023 | 13:42 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Monday, 2023-05-08 02:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 5 May 2023 | 03:36 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 4 May 2023 | 16:35 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 4 May 2023 | 15:39 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-05-05 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 4 May 2023 | 03:01 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Thursday, 2023-05-04 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 3 May 2023 | 15:48 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Thursday, 2023-05-04 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 3 May 2023 | 02:49 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-03 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 2 May 2023 | 12:53 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The current impacted product list has been updated. For GKE customers, a few GKE zonal clusters with a control plane in europe-west9-a continue to be unavailable. Our product team will be reaching out to affected GKE customers with additional information. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Wednesday, 2023-05-03 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. New instances are unaffected. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 2 May 2023 | 09:16 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The list of current impacted products has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-02 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 2 May 2023 | 03:42 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-02 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 1 May 2023 | 14:11 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-02 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 1 May 2023 | 11:06 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected data center. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Tuesday, 2023-05-02 04:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 30 Apr 2023 | 22:48 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Monday, 2023-05-01 11:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 28 Apr 2023 | 18:49 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Sunday, 2023-04-30 23:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 28 Apr 2023 | 06:47 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-04-28 19:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 27 Apr 2023 | 18:53 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Friday, 2023-04-28 07:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 27 Apr 2023 | 11:03 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expect to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. We will provide an update by Thursday, 2023-04-27 19:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access existing Cloud resources in the europe-west9-a zone. Workaround: Customers can failover to other zones in europe-west9 or to other regions. |
| 27 Apr 2023 | 06:39 PDT | Summary: Multiple Google Cloud services in the europe-west9-a zone are impacted Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. Impact is now limited to services in europe-west9-a. There is no ETA for full recovery of operations in europe-west9-a at this time. We expected to see extended outages for some services. Customers are advised to failover to other zones/regions if they are impacted. The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM). The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL We will provide an update by Thursday, 2023-04-27 11:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 27 Apr 2023 | 02:30 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted. The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE) The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: The issue with Cloud Pub/Sub has been resolved for all affected users as of Wednesday, 2023-04-26 16:07 S/Pacific. For more information please see here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Thursday, 2023-04-27 07:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 27 Apr 2023 | 01:54 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted. The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE) The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Thursday, 2023-04-27 03:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 26 Apr 2023 | 22:49 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted. The following services have fully recovered in europe-west9: Google Cloud Storage (GCS) Cloud Key Management Service (KMS) Cloud Identity and Access Management (IAM) Google Kubernetes Engine (GKE) The following services have recovered in europe-west9-b and europe-west9-c, but continue to be impacted in europe-west9-a: Google Compute Engine (GCE) Cloud Run Google Cloud Load Balancer (GCLB) DataProc Cloud SQL Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific. For more information please see here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Thursday, 2023-04-27 03:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 26 Apr 2023 | 19:52 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Description: Water intrusion in a datacenter in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Thursday, 2023-04-27 00:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 26 Apr 2023 | 16:01 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. Cloud Console: Experienced a global outage, which has been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Wednesday, 2023-04-26 20:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 26 Apr 2023 | 11:59 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. Cloud Console: Experienced global outages which have been mitigated. Management tasks should be operational again for operations outside the affected region (europe-west9). Primary impact was observed from 2023-04-25 23:15:30 PDT to 2023-04-26 03:38:40 PDT. GCE Control Plane: Experienced global outages, which have been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT and 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). Secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal Cloud Pub/Sub: For information related to ongoing Cloud Pub/Sub impact, please see the latest status here: https://status.cloud.google.com/incidents/j6LfsjxCXhVDjmGGPhS7#2c2sBHWU84yPDJ8y1ar4 BigQuery: For information related to ongoing BigQuery impact, please see the latest status here: https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG We will provide an update by Wednesday, 2023-04-26 16:00 US/Pacific, or upon any significant development. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions. |
| 26 Apr 2023 | 07:01 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 12:00 US/Pacific, or upon any significant development. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 06:58 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in a data center in europe-west9 has caused a multi-cluster failure and has led to a shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 12:00 US/Pacific, or upon any significant development. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 05:52 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. For management tasks Cloud Console should be operational again for operations outside the affected region (europe-west9). We will provide an update by Wednesday, 2023-04-26 07:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 05:10 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. For management tasks Cloud Console should be operational again for operations outside the affected region (europe-west9) We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 05:07 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 04:51 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 06:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions Customer should use gcloud commands instead of Cloud Console for management tasks |
| 26 Apr 2023 | 03:39 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 05:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Workaround: Customers can failover to zones in other regions Customer should use gcloud commands instead of Cloud Console for management tasks |
| 26 Apr 2023 | 02:42 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 04:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page Workaround: Customers can failover to zones in other regions Customer should use gcloud commands instead of Cloud Console for management tasks |
| 26 Apr 2023 | 02:25 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 03:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region. Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page Workaround: Customers can failover to zones in other regions Customer should use gcloud commands instead of Cloud Console for management tasks |
| 26 Apr 2023 | 01:50 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 03:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region Workaround: Customers can failover to zones in other regions |
| 26 Apr 2023 | 00:35 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted. Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 02:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region Workaround: Customers can failover to zones in other regions |
| 25 Apr 2023 | 23:05 PDT | Summary: Multiple Google Cloud services in the europe-west9 region are impacted Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted. We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9 region Workaround: Customers can failover to zones in other regions |
| 25 Apr 2023 | 22:21 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones if they are impacted. We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones |
| 25 Apr 2023 | 22:18 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted. We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones within europe-west9 |
| 25 Apr 2023 | 22:16 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted. We will provide an update by Wednesday, 2023-04-26 00:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other available zones |
| 25 Apr 2023 | 20:51 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone. There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted. We will provide an update by Tuesday, 2023-04-25 22:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones within europe-west9 |
| 25 Apr 2023 | 19:56 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-04-25 21:00 US/Pacific with current details. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones within europe-west9 |
| 25 Apr 2023 | 19:25 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-04-25 20:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones within europe-west9 |
| 25 Apr 2023 | 19:00 PDT | Summary: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Description: We are investigating an issue affecting multiple Cloud services in the europe-west9-a zone Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-04-25 19:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may be unable to access Cloud resources in europe-west9-a Workaround: Customers can fail over to other zones within europe-west9 |
- All times are US/Pacific