Mini Incident Report

We apologize for any inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 8 May, 2024 18:45

Incident End: 8 May, 2024 21:40

Duration: 2 hrs 55 minutes

Affected Services and Features:

• Persistent Disk,
• Google Cloud Dataflow
• Google Cloud Pub/Sub
• Google Big Query
• Google Compute Engine
• Cloud Filestore
• Virtual Private Cloud (VPC)
• Cloud SQL
• Cloud Logging
• Cloud Bigtable
• Cloud Apigee

Regions/Zones: australia-southeast1

Description:

Multiple Google Cloud products experienced service disruptions of varying impact and duration, with the longest lasting being 2 hours and 55 minutes in the australia-southeast1 region. From preliminary analysis, the root cause of this incident is currently believed to be an unplanned power event caused by a power failover due to a utility company outage. Google will complete a full Incident Report in the following days that will provide a detailed root cause.

Customer Impact:

• Persistent Disk - impacted users experienced slow or unavailable devices.
• Google Cloud Dataflow - impacted users experienced an increase in streaming jobs with watermarks in australia-southeast1-a zone for a duration of 30 minutes.
• Google Cloud Pub/Sub - users experienced an increased error rate for “Publish requests” for a duration of about 35 minutes.
• Google Big Query - impacted users experienced failures for BigQuery jobs in the australia-southeast1 region.
• Google Compute Engine - impacted VMs went into repair mode for about 45 minutes.
• Cloud Filestore - multiple Filestore instances in australia-southeast1-a were unavailable and had missing metrics for a duration of 2 hours 55 minutes, with the last impacted instance confirmed to have recovered at 21:43 PT.
• Virtual Private Cloud (VPC) - the impacted users experienced packet loss, unavailability of existing VMs and delays while creating new VMs.
• Cloud SQL - impacted users experienced errors when accessing their Cloud SQL database instances in the australia-southeast1-a zone.
• Cloud Logging - Cloud Logging experienced a minor increase in ingestion error in australia-southeast1 for a duration of 15 minutes.
• Cloud Bigtable - users experienced a high error rate in the impacted region for a duration of about 25 minutes.
• Cloud Apigee - impacted users received 5XX and 2XX error for a duration of 30 minutes.

Additional details:

After service mitigation and full closure of the incident, there was continued Persistent Disk impact for a narrowed group of customers identified. This has since been resolved with no further isolated impact.

The issue with Apigee, Cloud Filestore, Cloud Logging, Google BigQuery, Google Cloud Bigtable, Google Cloud Dataflow, Google Cloud Pub/Sub, Google Cloud SQL, Google Compute Engine, Persistent Disk, Virtual Private Cloud (VPC) has been resolved for all affected users as of Wednesday, 2024-05-08 21:40 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Multiple services impacted in australia-southeast1.

Description: We are experiencing an issue with Persistent Disk, Google Cloud Dataflow, Google Cloud Pub/Sub, Google BigQuery, Google Compute Engine, Cloud Filestore, Virtual Private Cloud (VPC), Cloud logging, Cloud SQL, Cloud Bigtable, Apigee beginning at Wednesday, 2024-05-08 18:45 US/Pacific.

Mitigation strategy has been identified. The services are now recovering.

We will provide an update by Wednesday, 2024-05-08 23:00 US/Pacific with current details.

Diagnosis: Multiple GCP services are experiencing issues in australia-southeast1 region.

Persistent Disk: While most devices have restored their functionality, some users might encounter slow or unavailable devices.

Google Cloud Dataflow: Users experienced issues for streaming jobs with Watermark increasing. The issue with Google Cloud Dataflow is mitigated at 2024-05-08 19:47:27 PDT.

Google Cloud Pub/Sub: The PubSub impact is mitigated.

Google BigQuery: The impacted users experienced failures with the bigquery jobs in the australia-southeast1 Region. The issue with Google Bigquery has been resolved for all the affected users as of Wednesday, 2024-05-08 21:13 US/Pacific.

Google Compute Engine: VM’s went into repair for around 45 minutes and have started recovering.

Cloud Filestore: The Filestore is partially recovered. However, a small subset of users would not able to access the NFS filestore in the australia-southeast1-a zone.

Virtual Private Cloud (VPC): The impacted users may face delays while creating new VMs and packet loss / unreachability for existing VMs.

Cloud SQL: A subset of the Cloud SQL users are experiencing errors when accessing their Cloud SQL database instances in the australia-southeast1-a zone.

Cloud logging: All requests are failing at the send request step. The issue with Cloud logging has been resolved for all the affected users as of Wednesday, 2024-05-08 21:16:07 US/Pacific.

Cloud Bigtable: Cloud Bigtable experienced a high error rate for 25 minutes in australia-southeast1-a due to a power event. The issue with Cloud Bigtable has been resolved for all the affected users as of Wednesday, 2024-05-08 20:08:30 US/Pacific.

Apigee: There was a minor outage due to the GKE error which caused all of the nodes to restart. The GKE cluster is currently undergoing repair. This resulted in a 30 minute outage for the customer. The issue with Apigee has been resolved for all the affected users as of Wednesday 2024-05-08 20:34:47 US/Pacific.

Workaround: None at this time.

Summary: Multiple services impacted in australia-southeast1.

Description: We are experiencing an issue with Big Query, Google filestore, Cloud PubSub beginning at Wednesday, 2024-05-08 18:45 US/Pacific.

Mitigation strategy has been identified. The services are now recovering.

We will provide an update by Wednesday, 2024-05-08 21:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Multiple GCP services are experiencing issues in australia-southeast1 region.

Persistent Disk: While most devices have restored their functionality, some users might encounter slow or unavailable devices.

Google Cloud Dataflow: Users experienced issues for streaming jobs with Watermark increasing. The issue with Google Cloud Dataflow is mitigated at 2024-05-08 19:47:27 PDT.

Google Cloud Pub/Sub: The PubSub impact is mitigated.

Google BigQuery: The impacted users may experience failures with the bigquery jobs in the australia-southeast1 Region.

Google Compute Engine: VM’s went into repair for around 45 minutes and have started recovering. The issue with the Compute Engine is mitigated at 2024-05-08 19:43:43 PDT.

Cloud Filestore: The impacted customers are unable to access the NFS Filestores in the australia-southeast1-a Zone.

Workaround: None at this time.