Mini Incident Report

We apologize for the inconvenience this outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 29 March 2025 12:53

Incident End: 29 March 2025 19:12

Duration: 6 hours, 19 minutes

Affected Services and Features:

• AlloyDB for PostgreSQL
• BigQuery
• Cloud Bigtable
• Cloud Composer
• Cloud Dataflow
• Cloud Filestore
• Cloud Firestore
• Cloud Identity and Access Management
• Cloud Interconnect
• Cloud Key Management Service
• Cloud Kubernetes Engine
• Cloud NAT
• Cloud Router
• Cloud SQL
• Cloud Spanner
• Cloud VPN
• Compute Engine
• Managed Service for Apache Kafka
• Memorystore for Redis
• Persistent Disk
• Secret Manager
• Virtual Private Cloud

Regions/Zones: us-east5-c

Description:

Multiple Google Cloud products were impacted in us-east5-c, with some zonal resources unavailable, for a duration of 6 hours and 19 minutes.

The root cause of the issue was a utility power outage in the zone and a subsequent failure of batteries within the uninterruptible power supply (UPS) system supporting a portion of the impacted zone. This failure prevented the UPS from operating correctly, thereby preventing a power source transfer to generators during the utility power outage. As a result, some Compute Engine instances in the zone experienced downtime. The incident also caused some packet loss within the us-east5-c zone, as well as some capacity constraints for Google Kubernetes Engine in other zones of us-east5. Additionally, a small number of Persistent Disks were unavailable during the outage.

Google engineers diverted traffic away from the impacted location to partially mitigate impact for some services that did not have zonal resource dependencies. Engineers bypassed the failed UPS and restored power via generator, allowing the underlying infrastructure to come back online. Impact to all affected Cloud services was mitigated by 29 March 2025 at 19:12 US/Pacific.

Google will complete a full Incident Report in the following days that will provide a detailed root cause analysis.

Customer Impact:

Customers experienced degraded service or zonal unavailability for multiple Google Cloud products in us-east5-c.

Additional details:

The us-east5-c zone has transitioned back to primary power without further impact as of 30 March 2025 at 17:30 US/Pacific.

Currently, the us-east5-c zone is stable on an alternate power source. All previously impacted products are mitigated as of 19:12 US/Pacific.

A small number of Persistent Disks remain still in recovery, and are actively being worked on. Customers still experiencing issues attaching Persistent Disks should open a support case.

Our engineers continue to monitor service stability prior to transitioning back to primary power.

We will provide continuing updates via PSH by Sunday, 2025-03-30 01:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Our engineers are actively working on recovery following a power event in the affected zone. Full recovery is currently expected to take several hours.

The impacted services include Cloud Interconnect, Virtual Private Cloud (VPC), Google Compute Engine, Persistent Disk, AlloyDB for PostgreSQL, Cloud Dataproc, Cloud Dataflow, Cloud Filestore, Identity and Access Management, Cloud SQL , Google Kubernetes Engine, Cloud Composer, BigQuery, Cloud Bigtable and more.

We have determined that no other zones (a, b) in the us-east5 region are impacted.

We will provide an update by Saturday, 2025-03-29 20:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.