Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google App Engine

[Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors.

Incident began at 2022-06-08 09:16 and ended at 2022-06-14 20:02 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Warsaw (europe-central2)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Zurich (europe-west6)Montréal (northamerica-northeast1)São Paulo (southamerica-east1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
15 Jun 2022 02:38 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 08 June 2022 09:16

Incident End: 14 June 2022 20:02

Duration: 6 day, 10 hours, 46 minutes

Affected Services and Features: Google Cloud App Engine, Google Cloud Functions.

Regions/Zones: Global

Description:

Google Cloud App Engine customers might have experienced elevated errors while using the Secrets Manager for a duration of 6 days, 10 hours and 46 minutes. From preliminary analysis, the root cause of the issue is, regression in a recent software release relating to Secret Manager functionality in Google Cloud App Engine.

Customer Impact:

Customers might have experienced increased errors from apps or functions that use the Secrets Manager.

Additional details:

The Secret Manager functionality related release updates are rolled back to mitigate the issue. During the impact customers were advised to mount each secret in a separate path as per https://cloud.google.com/functions/docs/configuring/secrets.

14 Jun 2022 21:03 PDT

The issue with Google App Engine has been resolved for all affected projects as of Tuesday, 2022-06-14 20:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

14 Jun 2022 17:36 PDT

Summary: [Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors.

Description: Mitigation work is still underway by our engineering team.

We will provide more information by Tuesday, 2022-06-14 22:00 US/Pacific.

Diagnosis: Customers using the Secrets Manager are experiencing increased errors.

Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

14 Jun 2022 14:28 PDT

Summary: [Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors.

Description: Mitigation work is still underway by our engineering team.

We will provide more information by Tuesday, 2022-06-14 18:00 US/Pacific.

Diagnosis: Customers using the Secrets Manager are experiencing increased errors.

Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

14 Jun 2022 09:37 PDT

Summary: Increase in errors correlated with binary rollout

Description: Mitigation work is currently underway by our engineering team.

The mitigation is completed completed for europe-west3. The mitigation ETA for us-central1 is Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update.

We will provide more information by Tuesday, 2022-06-14 14:30 US/Pacific.

Diagnosis: Customers using Secrets Manager are experiencing increased errors.

Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

14 Jun 2022 08:08 PDT

Summary: Increase in errors correlated with binary rollout

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to be completed for europe-west3 Tuesday, 2022-06-14 10:00 US/Pacific, and for us-central1 Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update.

We will provide more information by Tuesday, 2022-06-14 10:10 US/Pacific.

Diagnosis: Customers using Secrets Manager are experiencing increased errors.

Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

14 Jun 2022 06:12 PDT

Summary: Increase in errors correlated with binary rollout

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to be completed for europe-west3 Tuesday, 2022-06-14 10:00 US/Pacific, and for us-central1 Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update.

We will provide more information by Tuesday, 2022-06-14 08:27 US/Pacific.

Diagnosis: Customers are experiencing increased errors.

Workaround: None at this time.