Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Console

Elevated gcloud crashes for service account users

Incident began at 2023-05-02 14:45 and ended at 2023-05-02 17:19 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
4 May 2023 13:26 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 2 May 2023 14:45

Incident End: 2 May 2023 17:19

Duration: 2 hours, 34 minutes

Affected Services and Features: Google Cloud CLI

Regions/Zones: Global

Description:

Google Cloud CLI users experienced elevated errors for a period of 2 hours and 34 minutes.

Google engineering teams were alerted by internal monitoring probes and started to work on the mitigation immediately. From preliminary analysis, the root cause of the issue was an incorrect logic used for parsing the flag that was introduced as part of a configuration change. The issue was resolved by rolling back the configuration changes, preventing any further impact to the users.

Customer Impact:

Google Cloud CLI users using service accounts (non-human accounts), encountered the following error message:

“gcloud crashed (AttributeError): 'bool' object has no attribute 'lower'”

Users were able to mitigate this in three ways:

  • Disable auth/service_account_use_self_signed_jwt explicitly
  • Disable feature flags
  • Downgrade to a lower version of Google Cloud CLI from version 419.0.0

Additional details:

Impact was limited only to Google Cloud CLI version 419.0.0 and above, using service accounts.

Reference(s):

[1] https://stackoverflow.com/a/76159563
2 May 2023 18:04 PDT

The issue with Cloud Data Fusion, Cloud Memorystore, Cloud Spanner, Cloud Workflows, Dataplex, Eventarc, Google Cloud Dataproc, Google Cloud Deploy, Google Compute Engine, Google Kubernetes Engine, Media CDN has been resolved for all affected users as of Tuesday, 2023-05-02 17:19 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
2 May 2023 18:02 PDT

Summary: CEP probes failing in setup-sdk step

Description: We are experiencing an issue with Cloud Spanner, Cloud Workflows, Google Compute Engine, Cloud Data Fusion, Google Cloud Deploy, Google Cloud Dataproc, Cloud Memorystore, Eventarc, Dataplex, Google Kubernetes Engine, Media CDN beginning at Tuesday, 2023-05-02 14:45 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-05-02 18:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: The gcloud command crashes when parsing the feature flags triggered by a feature flag.

Workaround: A workaround to disable feature flag is: gcloud config set enable_feature_flags false.