Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Developer Tools, Google Kubernetes Engine, Artifact Registry, Google Cloud Storage, Google Cloud Composer, Identity and Access Management, Google App Engine

Multiple products may return error messages across Asia/Australia

Incident began at 2022-12-09 04:50 and ended at 2022-12-09 07:55 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)

Date Time Description
9 Dec 2022 15:11 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 09 December 2022 04:50

Incident End: 09 December 2022 07:55

Duration: 3 hours, 5 minutes

Affected Services and Features:

  • Artifact Registry: New repository creations
  • Google Cloud Composer: Composer environment creations
  • Google Cloud Storage: Operations that check IAM policies, especially CreateBucket
  • Google Kubernetes Engine: New cluster creation
  • Identity and Access Management: Not directly affected, only downstream services.

Regions/Zones: australia-southeast1, asia-east1, asia-east2, asia-northeast1, asia-northeast2, asia-northeast3, asia-south1, asia-south2, asia-southeast1, asia-southeast2, australia-southeast2, and asia (Google Cloud multi-region).

Description:

Google Cloud Identity and Access Management (IAM) in the Asia multi-region experienced unavailability, which impacted several downstream Google Cloud services for a period of 3 hours and 17 minutes. Customers directly using IAM were not affected. The root cause was related to a configuration change with unintended consequences, which was immediately rolled back, once identified as the problem, thereby restoring services.

Customer Impact:

Artifact Registry

  • Affected customers would not have been able to create new repositories in the Asia multi-region. Customers would have received a failure message of “Request contains an invalid argument.”

Google Cloud Composer

  • Affected customers in the Asia multi-region were not able to create Composer environments, receiving errors indicating either an issue with AppEngine or Google Kubernetes Engine (GKE) cluster creation.

Google Cloud Storage

  • Affected customers would have experienced increased error rates when creating buckets in the Asia multi-region. GKE uses Google Cloud multi-regional buckets internally, so regional GKE customers throughout Asia Pacific were affected.

Google Kubernetes Engine

  • Affected customers would have been unable to create new clusters in the affected regions.
9 Dec 2022 08:22 PST

The issue with Artifact Registry, Google App Engine, Google Cloud Composer, Google Cloud Storage, Google Kubernetes Engine, Identity and Access Management has been resolved for all affected projects as of Friday, 2022-12-09 08:06 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
9 Dec 2022 07:56 PST

Summary: Multiple products may return error messages across Asia/Australia

Description: We are experiencing an issue with Identity and Access Management, Identity and Access Management, Identity and Access Management, Identity and Access Management, Google Cloud Composer, Identity and Access Management, Identity and Access Management, Identity and Access Management, Google Cloud Composer, Identity and Access Management, Google Cloud Composer, Google Cloud Composer, Google Cloud Composer, Google Cloud Composer, Google Cloud Composer, Google Cloud Composer, Identity and Access Management, Google Cloud Composer, Identity and Access Management, Google Cloud Composer, Identity and Access Management, Google Cloud Composer, Identity and Access Management, Identity and Access Management, Google Cloud Composer, Artifact Registry, Identity and Access Management, Google Cloud Composer, Artifact Registry, Google Kubernetes Engine, Identity and Access Management, Artifact Registry, Identity and Access Management, Google Cloud Composer, Artifact Registry, Google Kubernetes Engine, Google App Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-12-09 08:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Multiple products may return error messages

Workaround: None at this time.