Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Compute Engine, Google Cloud DNS, Google Cloud Networking, Google Cloud Composer, Google App Engine, Service Directory

Google Cloud Infrastructure is failing to push cloud configs

Incident began at 2022-05-08 23:24 and ended at 2022-05-09 02:42 (all times are US/Pacific).

Previously affected location(s)

Global

Date Time Description
10 May 2022 15:40 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 08 May 2022 23:24

Incident End: 09 May 2022 02:42

Duration: 3 hours, 18 minutes

Affected Services and Features:

  • Google Cloud Networking
  • Google App Engine
  • Google Cloud Composer

Regions/Zones: Global

Description:

Multiple Google Cloud services experienced issues from 8 May 2022 23:24 to 9 May 2022 02:42 US/Pacific. From preliminary analysis, the root cause of the issue was network issues that caused tests to fail which led the configuration management system to treat the config as a bad config and prevented propagating configuration changes to Google Cloud infrastructure backend servers.

Customer Impact:

  • Google Cloud Networking - Customers modifying backend configuration for load balanced products or health check configuration may have seen their configuration accepted but not propagate, or fail after a timeout
  • Google App Engine - Customers may have observed the error, DEADLINE_EXCEEDED - “Timed out waiting for the app infrastructure to become healthy” when attempting to deploy updates.
  • Google Cloud Composer - Customers would have been unable to create Composer 1 and some Composer 2 "Private IP" environments (only those using VPC peering, environments using Private Service Connect were unaffected).
9 May 2022 02:49 PDT

The issue with Google App Engine, Google Cloud Composer, Google Cloud DNS, Google Cloud Networking, Google Compute Engine, Service Directory has been resolved for all affected users as of Monday, 2022-05-09 02:42 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

9 May 2022 02:27 PDT

Summary: Google Cloud Infrastructure is failing to push cloud configs

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2022-05-09 04:00 US/Pacific.

Diagnosis: Customers may experience the following: 1) External HTTP/S Load Balancing, Cloud CDN - config changes will be accepted but fail to propagate 2) Cloud Armor rules may also be affected 3) App Engine Flex customers may see apps fail to deploy 4) Traffic Director may see configs fail to propagate 5) VM instance group health checks & functionality like autoscaling/autohealing may be affected.

Product Impact: Google App Engine Flex: Impact/Diagnosis: Users may observe error, DEADLINE_EXCEEDED. “Timed out waiting for the app infrastructure to become healthy”. This occurs when attempting to deploy major version updates. Workaround: None.

Google Cloud Networking Impact/Diagnosis: Cloud customers modifying backend configuration for load balanced products, or health check configuration, may see their configuration accepted but not propagate, or fail after a timeout. Workaround: None

Cloud Composer: Impact: Users are unable to create Composer 1 and Composer 2 "Private IP" environments. Workaround: No workaround is available for Composer 1. For Composer 2 “Private IP” the workaround is to create environments with Private Service Connect. Composer 2 “Public IP” environment creations should be successful.

Workaround: None

9 May 2022 01:28 PDT

Summary: GICS is failing to push cloud configs

Description: We are experiencing an issue with Google Cloud DNS, Service Directory beginning at Sunday, 2022-05-08 23:24 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2022-05-09 03:00 US/Pacific with current details.

Diagnosis: Customer's might experience following:

  1. External HTTP/S Load Balancing, Cloud CDN - config changes will be accepted & fail to propagate
  2. Cloud Armor rules may also be affected
  3. Appengine flex customers may see apps fail to deploy
  4. Traffic Director may see configs fail to propagate
  5. VM instance group healthchecks & functionality like autoscaling/autohealing may be affected

Workaround: None