Incident affecting Google Cloud Bigtable, Data Catalog, Google Cloud Pub/Sub, Google Cloud SQL
Multiple Cloud products are experiencing elevated error rates in us-east5 due to authentication failures
Incident began at 2023-06-26 10:15 and ended at 2023-06-26 11:04 (all times are US/Pacific).
Previously affected location(s)
| ||26 Jun 2023||16:27 PDT|| |
Mini Incident Report
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.
(All Times US/Pacific)
Incident Start: 26 June 2023 at 10:15
Incident End: 26 June 2023 at 11:04
Duration: 49 minutes
Affected Services and Features:
CloudSQL Cloud Pub/Sub Pub/Sub Lite Cloud Bigtable Data Catalog Google Compute Engine Cloud Spanner Google Cloud Storage
Google Cloud Identity and Access Management experienced elevated error rates that resulted in authentication failures for multiple Google Cloud services in the us-east5 region for a duration of 49 minutes. The trigger for this outage was a new deployment for some Cloud IAM workloads in the us-east5 region. The new workloads were incorrectly configured to use authentication keys whose corresponding verification keys were not yet distributed. As Cloud services detected the new IAM workloads and started to send them authentication requests, they received in return authentication proof tokens that could not be verified, resulting in failed workflows. The issue was mitigated by redirecting traffic from the affected IAM workloads.
Google Compute Engine
Google Cloud Storage
| ||26 Jun 2023||11:37 PDT|| |
The issue with Data Catalog, Google Cloud Bigtable, Google Cloud Pub/Sub, Google Cloud SQL, Cloud Spanner has been resolved for all affected users as of Monday, 2023-06-26 11:06 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||26 Jun 2023||11:28 PDT|| |
Summary: Multiple Cloud products are experiencing issues in us-east5
Description: We are experiencing an issue with Google Cloud SQL, Google Cloud Bigtable, Data Catalog, Google Cloud Pub/Sub.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2023-06-26 12:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: None at this time.
Workaround: None at this time.
- All times are US/Pacific