Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 6 May 2023 06:05

Incident End: 6 May 2023 10:35

Duration: 4 hours, 30 minutes

Affected Services and Features:

Cloud Interconnect, Cloud Pub/Sub, Cloud KMS, Google Cloud Bigtable, Google Cloud Storage, Cloud SQL, Google Compute Engine, Cloud Filestore, Google Kubernetes Engine, Persistent Disk, Memorystore, Cloud Networking, Virtual Private Cloud (VPC), Cloud Dataflow

Regions/Zones: europe-west9-c, europe-west9

Description:

Multiple Google Cloud products in the europe-west9-c zone were unavailable to customers due to a cooling issue in the data center for a duration of 4 hours and 30 minutes. Cooling was temporarily interrupted due to an electrical issue with the cooling control system. To protect the physical machines from damage, the machines were proactively taken offline while cooling was restored.

Customer Impact:

Europe-west9:

• Cloud Interconnect - Affected customers experienced create/delete operations failing and packet loss to/from endpoints.
• Cloud Pub/Sub - Affected customers experienced missing backlog stats data and message backlog unavailable for messages greater than 10 days old.
• Cloud KMS (Key Management System) - External Key Management (EKS) returned errors with elevated errors for fetching.
• Google Cloud Storage - Affected customers experienced elevated write errors.
• Cloud SQL - Non-HA instances were unavailable.
• Cloud Networking - Virtual Machines were unreachable.
• Cloud Dataflow - Streaming jobs were stuck along with batch jobs as the VMs were down.

Europe-west9-c:

• Google Cloud Bigtable - Admin operations were unavailable.
• Google Compute Engine - Virtual Machines (VMs) were unavailable during the incident and customers were * unable to create/delete VMs.
• Cloud Filestore - Affected customers experienced some unavailability due to cluster shutdown.
• Google Kubernetes Engine - Some Regional and Zonal clusters were unavailable during the incident.
• Persistent Disk - All zonal customers were impacted as Google Compute Engine was unavailable.
• Memorystore - Standard tier instances were impacted with replica nodes unresponsive.
• Virtual Private Cloud - Affected customers experienced packet loss to/from endpoints and admin (create/delete/modify) operations failed.

The issue with all affected Google Cloud Products has been resolved for all affected users as of Saturday, 2023-05-06 11:40 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 12:00 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 11:30 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 11:30 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 11:00 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 11:00 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 09:45 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide an update by Saturday, 2023-05-06 09:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Cloud products beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 09:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Cloud products beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 08:45 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Cloud products beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 08:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers Google Cloud Bigtable users can read/write but admin operations like create/update/delete instances are not operational.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Cloud products beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 08:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers. Google Cloud Bigtable users can read/write but can't create/update/delete instances.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Compute Engine beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 08:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers. Google Cloud Bigtable users can read/write but can't create/update/delete instances.

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Compute Engine beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 07:45 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time

Summary: Multiple Cloud products impacted in europe-west9-c zone

Description: We are experiencing an issue with Google Compute Engine beginning on Saturday, 2023-05-06 06:40 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-05-06 07:15 US/Pacific with current details.

Diagnosis: Google Cloud products in europe-west9-c may not be available to the customers

Workaround: None at this time