Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Composer, Google App Engine

We are experiencing issues in App Engine Flexible and Cloud Composer

Incident began at 2022-11-08 18:59 and ended at 2022-11-08 20:25 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Warsaw (europe-central2)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Zurich (europe-west6)Montréal (northamerica-northeast1)São Paulo (southamerica-east1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
9 Nov 2022 14:48 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

Affected Services and Features:

Google App Engine - App Engine Flexible Google Cloud Composer

Regions/Zones: global

Description:

Google App Engine (GAE) Flexible and Google Cloud Composer operations (i.e. create, update, delete operations) had intermittent failures for customers for a total duration of 10 hours, 30 minutes globally.

From preliminary analysis, the root cause of the issue wass slow response to deployment workflows from one of our data centers hosting a backend component that manages deployments.

The first two occurrences were self-solved by our automatic processes. While the investigations continued, the issue recurred multiple times. and the potential cause was identified and the traffic was directed away from the slow responding data center mitigating the issue on 09 November 2022 at 08:37 US/Pacific.

Detailed information on occurrences:

(All Times US/Pacific)

First Occurrence :

 Impact Start: 08 November 2022 14:15
 Impact End:  08 November 2022 15:05
 Impact Duration: 50 minutes

Second Occurrence :

 Impact Start: 08 November 2022 16:40
 Impact End:   08 November 2022 19:37
 Impact Duration: 2 hours, 57 minutes

Third Occurrence :

 Impact Start: 09 November 2022 01:44
 Impact End:   09 November 2022 08:27
 Impact Duration: 6 hour, 43 minutes

Total Duration: 10 hours, 30 minutes

Customer Impact:

  • Google App Engine Flexible : 16% of GAE flexible projects may have experienced issues with create, update, delete operations.
  • Cloud Composer : Composer environment creation operations may have failed, typically with an error that mentions App Engine.
8 Nov 2022 20:25 PST

The issue with Google App Engine, Google Cloud Composer has been resolved for all affected projects as of Tuesday, 2022-11-08 20:20 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
8 Nov 2022 20:15 PST

Summary: We are experiencing issues in App Engine Flexible and Cloud Composer

Description: We are experiencing an issue with Google App Engine, Google Cloud Composer.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2022-11-08 22:03 US/Pacific with current details.

Diagnosis: App Engine Flexible : Customers may experience App Engine Flexible deployment issues with create/update/delete operations. Cloud Composer : Composer environment creation operations may fail, typically with an error that mentions App Engine

Workaround: Cloud Composer : Customers can retry the operation after it fails, although there is a chance it may fail again.
8 Nov 2022 19:27 PST

Summary: App Engine Flexible deployment failures

Description: We are experiencing an issue with Google App Engine beginning at Tuesday, 2022-11-08 14:15 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2022-11-08 20:55 US/Pacific with current details.

Diagnosis: Customers may experience App Engine Flexible deployment issues with create/update/delete operations.

Workaround: None