Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Bigtable, Data Catalog, Google Cloud Pub/Sub, Google Cloud SQL

Multiple Cloud products are experiencing elevated error rates in us-east5 due to authentication failures

Incident began at 2023-06-26 10:15 and ended at 2023-06-26 11:04 (all times are US/Pacific).

Previously affected location(s)

Columbus (us-east5)

Date Time Description
26 Jun 2023 16:27 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 26 June 2023 at 10:15

Incident End: 26 June 2023 at 11:04

Duration: 49 minutes

Affected Services and Features:

CloudSQL Cloud Pub/Sub Pub/Sub Lite Cloud Bigtable Data Catalog Google Compute Engine Cloud Spanner Google Cloud Storage

Regions/Zones: us-east5

Description:

Google Cloud Identity and Access Management experienced elevated error rates that resulted in authentication failures for multiple Google Cloud services in the us-east5 region for a duration of 49 minutes. The trigger for this outage was a new deployment for some Cloud IAM workloads in the us-east5 region. The new workloads were incorrectly configured to use authentication keys whose corresponding verification keys were not yet distributed. As Cloud services detected the new IAM workloads and started to send them authentication requests, they received in return authentication proof tokens that could not be verified, resulting in failed workflows. The issue was mitigated by redirecting traffic from the affected IAM workloads.

Customer Impact:

Cloud SQL

  • 1 failed update operation resulting in downtime for HA instance
  • 4 failed creations
  • Missing metrics for up to 50 minutes by <150 instances

Cloud Pub/Sub

  • Affected customers experienced elevated authentication errors.

Pub/Sub Lite

  • Affected customers experienced elevated authentication errors.

Cloud Bigtable

  • Affected customers experienced elevated 401 authentication errors.

Data Catalog

  • Affected customers experienced elevated 401 authentication errors.

Google Compute Engine

  • Affected customers experienced elevated 401 authentication errors.
  • About 50% of users were impacted in us-east5-c (300 QPS of 600) from 10:21 -11:05 a.m.

Cloud Spanner

  • Affected customers experienced elevated 401 authentication errors.

Google Cloud Storage

  • -Affected customers experienced elevated 400 authentication errors.
  • <1% of customers were affected from 10:20 - 11:09 a.m.
  • <1% QPS of total traffic received errors during this time peaking at around 1100 QPS before the incident was mitigated.
26 Jun 2023 11:37 PDT

The issue with Data Catalog, Google Cloud Bigtable, Google Cloud Pub/Sub, Google Cloud SQL, Cloud Spanner has been resolved for all affected users as of Monday, 2023-06-26 11:06 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
26 Jun 2023 11:28 PDT

Summary: Multiple Cloud products are experiencing issues in us-east5

Description: We are experiencing an issue with Google Cloud SQL, Google Cloud Bigtable, Data Catalog, Google Cloud Pub/Sub.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2023-06-26 12:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: None at this time.

Workaround: None at this time.