Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 7 November 2023 11:51

Incident End: 7 November 2023 15:46

Duration: 3 hours, 55 minutes

Affected Services and Features:

• Google Cloud Networking
• Virtual Private Cloud
• Google Compute Engine
• Persistent Disk

Regions/Zones: us-east4-a

Description:

Multiple Google Cloud products experienced availability issues for a duration of 3 hours, 55 minutes in us-east4-a. The issue stemmed from a power loss event, resulting in the loss of primary power to the affected racks and a failed transition to backup power. Services recovered once power was restored, and the hardware came back online.

Customer Impact:

Virtual Private Cloud

• Approximately 13% of hosts in us-east4-a were impacted.
• Customer VMs and workloads may have been interrupted due to underlying power failures.
• Customer VMs and workloads re-scheduled onto remaining available machines.
• Impact was fully mitigated at 12:38 US/Pacific.

Persistent Disk

• Approximately 1.6% of PD disks were impacted in us-east4-a at peak.
• Regional PDs were not affected.
• Customer devices experienced multi-second to multi-minute stuck I/Os including both reads and writes.
• Customer VMs may have panicked or returned I/O errors as a result of stuck I/Os.
• Impact was partially mitigated to 0.10% of affected devices at 12:16 US/Pacific and fully mitigated at 14:10 US/Pacific.

Google Compute Engine

• Approximately 12% of hosts in us-east4-a experienced availability issues.
• Impact was mostly mitigated at 15:46 US/Pacific.
• Google engineering is focusing efforts to address long tail impact affecting less than 1% of customers using - - Local SSDs, which could not be immediately restarted on alternate hosts and working directly with those customers to resolve.

The issue with Virtual Private Cloud is fully resolved at 12:38 US/Pacific and the issue with Persistent Disk is full resolved at 14:42 US/Pacific

The issue with Google Compute Engine is resolved for most of the affected users and our teams are working directly with the customers with remaining impact.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Compute Engine, Persistent Disk, Virtual Private Cloud are experiencing issues in us-east4-a

Description: We believe the issue with Virtual Private Cloud is fully resolved at 12:38 US/Pacific.

Persistent Disk impact is resolved except for a handful of disks which continue to have some stuck I/Os. The engineering team is aware of these disks and is working to repair them.

The issue with Google Compute Engine is resolved for most of the affected users and our teams are working directly with the customers with remaining impact.

We will provide an update by Tuesday, 2023-11-07 14:45 US/Pacific with current details.

Diagnosis: Persistent Disk:

• Multi-second to multi-minute stuck I/Os including both reads and writes.
• VMs may panic or return I/O errors as a result of stuck I/Os

Google Compute Engine:

• Customer VMs may experience availability issues and could show up in “repair” status in Cloud Console.

VPC:

• Customer VMs and workloads may have been interrupted.

Workaround:

Persistent Disk:

• Use a different zone for any zonal workloads that can be moved. Regional PDs should not be impacted.

Summary: Google Compute Engine, Persistent Disk, Virtual Private Cloud are experiencing issues in us-east4-a

Description: We are experiencing an issue with Persistent Disk, Virtual Private Cloud (VPC), Google Compute Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-11-07 15:02 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

Persistent Disk:

• Multi-second to multi-minute stuck I/Os including both reads and writes.
• VMs may panic or return I/O errors as a result of stuck I/Os

Workaround:

Persistent Disk: Use a different zone for any zonal workloads that can be moved. Regional PDs should not be impacted.

Summary: Google Persistent Disk customer may experience high read and write I/O latency in us-east4-a

Description: We are experiencing an issue with Persistent Disk beginning at Tuesday, 2023-11-07 11:51 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-11-07 15:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: * Multi-second to multi-minute stuck I/Os including both reads and writes.

• VMs may panic or return I/O errors as a result of stuck I/Os

Workaround: Use a different zone for any zonal workloads that can be moved. Regional PDs should not be impacted.

Summary: Google Persistent Disk customer may experience high read and write I/O latency in us-east4-a

Description: We are experiencing an issue with Persistent Disk beginning at Tuesday, 2023-11-07 11:51 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-11-07 14:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: None at this time.

Workaround: None at this time.