Service Health
For incidents related to Google Security Products, visit https://status.cloud.google.com/security. For incidents related to Looker (original), visit https://status.cloud.google.com/looker.
Incident affecting AlloyDB for PostgreSQL, Apigee, BigQuery Data Transfer Service, Cloud Data Fusion, Cloud Filestore, Cloud Firestore, Cloud Load Balancing, Cloud Memorystore, Cloud NAT, Cloud Run, Cloud Spanner, Cloud Workstations, Datastream, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud NetApp Volumes, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Google Kubernetes Engine, Looker (Google Cloud core), Managed Airflow, Managed Service for Apache Kafka, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Memorystore for Valkey, Network Connectivity Center, Persistent Disk, Private Service Connect, Speech-to-Text, Storage Transfer Service, VMWare engine, Virtual Private Cloud (VPC)
Multiple services were experiencing issues in the europe-west2 region.
Incident began at 2026-10-19 23:00 and ended at 2026-10-20 05:08 (all times are US/Pacific).
Previously affected location(s)
London (europe-west2)
| Date | Time | Description | |
|---|---|---|---|
| | 27 Oct 2026 | 08:13 PDT | Incident ReportSummaryOn Monday, 19 Oct 2026 (Tuesday, 20 Oct BST), a data center facility that hosts europe-west2-b, experienced an external electrical power disruption. This resulted in service unavailability for workloads located in this zone. As traffic was redirected to alternative cloud regions or unaffected zones within europe-west2 cloud region, these workloads experienced transient latency and elevated errors affecting services across europe-west2. The primary impact was mitigated after 6 hours 8 minutes. Root CauseFacility architecture and background: The data center facility in the UK region hosts Cloud clusters serving the europe-west2-b zone, which form part of the multi-zonal europe-west2 Cloud region. The site relies on a 2N utility power architecture—meaning the facility has complete duplicate power paths for every critical load—backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC). Primary utility power is supplied to the facility via two independent feeders (A and B), supplemented by a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events. The two high-voltage primary feeders terminate at separate primary intake switchboards capable of closed-loop operation if either board is taken offline. Switching between the primary and standby utility providers utilizes open transitions and key interlocks to prevent parallel connections across sources. Static Uninterruptible Power Supply (UPS) battery systems provide 5 minutes of autonomous runtime. This bridge power ensures uninterrupted operation while the PLC detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load. Prior to the utility disruption, the facility had been placed in a temporary single-feed (1N) bridged maintenance posture on the primary utility. Circuit breaker feeder B was physically opened and key-interlocked (Locked Out Tagged Out) to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026. Consequently, primary utility feed A was supporting the total critical load while repairs were conducted on the B feeder pathway. Root Cause: On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 06:53 BST), the primary utility provider experienced an outage on primary utility feed A, caused by an external underground cable fault on the utility network outside the data center facility. When the primary utility feed A was lost, the facility’s generator plant failed to receive a signal to automatically start and take critical load due to a design voltage-sensing and misconfiguration in the PLCs. The facility's static UPS battery banks carried the critical load as designed until their battery reserves depleted resulting in a complete power loss across data halls. The investigation of the PLC configuration identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed B breaker. The PLC was programmed to verify whether voltage was present on the feeder before commanding a generator start. As the utility feed B remained energized upstream, both PLCs sensed live upstream voltage on the disconnected path. As a result, the PLC logic indicated that a healthy utility source was available, even though the primary and secondary distribution feeds were de-energized, and did not issue a start command to the generator plant.; The onsite facilities team attempted to execute a remote load transfer via the PLC; however, the system inhibited the action. Recognizing that the UPS battery reserves would be exhausted before further remote operations could be resolved, the team transferred the facility load to the standby utility source, restoring the critical load that was lost once batteries depleted during this process. Once utility power was restored via the standby utility source, and UPS batteries started recharging, the onsite facility teams and OEM vendors reconfigured and validated PLC functionality on both undervoltage and breaker status. After the power loss was initially detected, Google engineers began work to mitigate failover failures and prepare service recovery. Upon restoration of the utility source, UPS and generator capabilities, they began the process to re-enable services in europe-west2-b. Remediation and PreventionThe sequence of events leading to customer impact was as follows:
Google is committed to preventing a repeat of this issue in the future and is completing the following actions: Data Center Action Items:
Cloud Service Action Items:
Detailed Description of ImpactBetween Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 Oct 2026, 05:08 US/Pacific (13:08 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 08 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (07:00 to 12:29 BST) (duration 5 hours, 29 minutes), customers experienced unavailability of zone-scoped resources in products, including:
The unavailability of those resources scoped to a zone that undergoes disruption is working within the design of a zonally-scoped system. Requests made to these resulted either in a timeout or error code during the disruption. Any workloads and resources hosted entirely within europe-west2-b became unreachable due to this event. Google Compute Engine's zonal resources such as VMs require customers to provision failover or cross-zone recovery resources if needed, and to set up a recovery mechanism that matches their needs. Some failovers were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism. Following platform service restoration, some customer workloads running on Compute Engine instances as well as products that rely on Compute Engine experienced a longer recovery. By 04:46 US/Pacific (12:46 BST) new Virtual Machine creation resumed and by 05:08 US/Pacific (13:08 BST) over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that were experiencing Virtual Machine problems were directed to file a customer support case as per normal procedure. While the physical power outage was isolated strictly to data center facilities in zone europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes of impact outside of zone europe-west2-b were: 1) Instantaneous demand surges on neighboring zones (a and c) and 2) Control-plane dependencies across zonal boundaries Affected products included: AlloyDB, Apigee Infra and Apigee Edge, BigQuery, Cloud Dataflow, Cloud Pub/Sub, Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Looker, Regional Persistent Disk, Vertex AI Prediction, and Cloud Speech-to-Text. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts. Customers would have experienced the inability to create, update, or delete some regional resources during this period. This was not a full regional control-plane outage, but certainly a degraded state. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out. Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. We did not identify any impact of the outage outside of the europe-west2 region. The following Google Cloud Platform services experienced impact within europe-west2:
|
| | 22 Oct 2026 | 01:42 PDT | Preliminary Incident ReportWe recognize that Google Cloud services are critical to your organization, and we sincerely apologize for the disruption this incident may have caused to your business. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. Date/Time of the IssueIncident Start:19 Oct 2026 23:00 US/Pacific / 20 Oct 2026 07:00 BST Incident End: 20 Oct 2026 05:10 US/Pacific / 20 Oct 2026 13:10 BST Duration: 6 hours, 10 minutes SummaryOn Monday, 19 October 2026 (Tuesday, 20 October BST), multiple Google Cloud Platform services in the europe-west2 region experienced service disruptions, elevated error rates, Application Programming Interface (API) timeouts, and processing latency for a duration of 6 hours and 10 minutes. The disruption was triggered by an electrical power interruption at the datacenter facility hosting availability zone europe-west2-b, which temporarily took physical host servers, network routing hardware, and storage systems offline. This subsequently caused secondary failover and queuing delays across unimpacted regional infrastructure. Preliminary Root CauseThe incident was triggered when a power grid disruption occurred at the datacenter facility hosting availability zone europe-west2-b. To maintain continuous power during the disruption, automatic transfer to facility backup power systems was triggered immediately. The facility load was transferred to the Uninterruptible Power Supply (UPS) battery banks during the start up sequence of on-site emergency diesel generators. The Programmable Logic Controller (PLC) automation system (both primary and backup PLCs) failed to initiate the logical automatic start/transfer operations of the generators, preventing backup generators from assuming the load. When UPS battery reserves were depleted, physical host servers, network switching fabric, and local storage arrays dedicated to europe-west2-b lost power and went offline. The sequence of events leading to customer impact was as follows:
Google engineers have begun a full root cause analysis (RCA) and will provide additional information once it is available. RemediationGoogle engineers were alerted to the disruption via automated monitoring systems and probers at 23:08 US/Pacific on 19 Oct (20 Oct, 07:08 BST), which detected elevated API error ratios, job submission timeouts, and loss of infrastructure reachability in europe-west2-b. Engineering teams immediately mobilized to isolate the affected infrastructure and commence recovery:
By 05:10 US/Pacific on Tuesday, 20 Oct(13:10 BST), service availability had recovered for the vast majority of customer workloads, and resolution for most of the customers across Google Cloud Platform services in the region was confirmed. Following the recovery of a small tail of remaining isolated workloads, the closure notification for the incident was posted by 07:03 US/Pacific (15:03 BST). Description of ImpactBetween Monday, 19 Oct 2026, 23:00 US/Pacific (Tuesday, 20 Oct 2026, 07:00 BST) and Tuesday, 20 October 2026, 05:10 US/Pacific (13:10 BST), customers utilizing Google Cloud Platform services in the europe-west2 region experienced service disruptions across two consecutive phases, lasting a total duration of 6 hours, 10 minutes. During the initial phase from 23:00 to 04:29 US/Pacific (07:00 to 12:29 BST) (duration 5 hours, 29 minutes), customers experienced elevated error rates, API timeouts, and processing delays until most Google Cloud Platform services recovered. Following platform service restoration, some customer workloads running on Compute Engine instances experienced a longer recovery tail. By 04:46 US/Pacific (12:46 BST) new Virtual Machine creation resumed and by 05:10 US/Pacific (13:10 BST) over 99% of Compute Engine Virtual Machines were in a healthy state. Any remaining customers that were experiencing Virtual Machine problems were directed to file a customer support issue as per normal procedure. While the physical power outage was isolated strictly to datacenter facilities in europe-west2-b, some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact. The primary causes were 1) instantaneous demand surges on unimpacted compute capacity and 2) synchronous control-plane validation dependencies across zonal boundaries. Affected products included: BigQuery, Looker, Google Compute Engine, Google Kubernetes Engine, Regional Persistent Disk, and Cloud SQL. These services experienced unexpected degradation and failover issues driven by sudden compute capacity exhaustion from surging demand, and control-plane API timeouts waiting on validations from the offline zone. Workloads and resources hosted entirely within europe-west2-b became unreachable. Data storage services with failover capability such as Cloud SQL, AlloyDB, Memorystore for Redis, and Cloud Bigtable failed over into nearby unimpacted zones during the europe-west2-b outage. Multi-zonal and regional services that spanned across the europe-west2 region experienced elevated latency, transient API errors (such as HTTP 500 and 503 errors), and task admission queuing as failover traffic concentrated onto unimpacted regional resources and cross-zone calls to offline components timed out. Storage write operations in the region experienced a transient error rate during the initial failover period, and regional query admission queues temporarily peaked before rapidly subsiding as capacity re-balanced. The following Google Cloud Platform services experienced impact:
|
| | 20 Oct 2026 | 07:03 PDT | Summary Multiple services were experiencing issues in the europe-west2 region. Description The issue with impacted GCP products has been mitigated for most of the customers and it is believed to be affecting currently a very small number of customers and our Engineering Team is working on it. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we're working on resolving the issue. Symptoms Google Cloud services were unavailable in europe-west2-b zone; some services in europe-west2 region might experienced elevated errors and intermittent network issues. Workaround None as the issue has been mitigated. |
| | 20 Oct 2026 | 05:44 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are seeing the recovery of services for most workloads in europe-west2 as of Tuesday, 2026-10-20 05:10 PDT. We will provide an update by Tuesday, 2026-10-20 07:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region). |
| | 20 Oct 2026 | 04:58 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. A power event caused an outage within europe-west2-b zone. The power event is mitigated. We are seeing the recovery of services and are prioritizing the recovery of existing workloads.Customers may not be able to yet create new workloads. Full recovery of existing workloads should be completed, tentatively by Tuesday, 2026-10-20 06:00. We will provide an update by Tuesday, 2026-10-20 06:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region). |
| | 20 Oct 2026 | 04:12 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Due to a power event we had suffered an outage within europe-west2-b zone. The power event is mitigated. Partial mitigation to infrastructure services will be observed within the next 30 minutes and the remaining services will be mitigated there after. We will provide an update by Tuesday, 2026-10-20 05:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Google Cloud services are unavailable in europe-west2-b zone; some services in europe-west2 region may experience elevated errors and intermittent network issues. Workaround If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region). |
| | 20 Oct 2026 | 04:00 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 05:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Note: Looker Studio is not impacted by this outage. Workaround If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region). |
| | 20 Oct 2026 | 02:24 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 04:00 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Note: Looker Studio is not impacted by this outage. Workaround If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region). |
| | 20 Oct 2026 | 01:35 PDT | Summary Multiple services are experiencing issues in the europe-west2 region. Description We are experiencing an issue with multiple services in europe-west2 beginning on Monday, 2026-10-19 23:08 PDT. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2026-10-20 02:30 PDT with current details. We apologize to all who are affected by the disruption. Symptoms Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region. Workaround None for now |
- All times are US/Pacific