Service Health
-
-
-
-
Incident affecting Google Compute Engine, Google Cloud DNS, Google Cloud Networking, Google Cloud Composer, Google App Engine, Service Directory, Cloud Load Balancing, Cloud CDN, Traffic Director, Virtual Private Cloud (VPC)
Google Cloud Infrastructure is failing to push cloud configs
Incident began at 2022-05-08 23:24 and ended at 2022-05-09 02:42 (all times are US/Pacific).
Previously affected location(s)
Global
| Date | Time | Description | |
|---|---|---|---|
| | 10 May 2022 | 15:40 PDT | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 08 May 2022 23:24 Incident End: 09 May 2022 02:42 Duration: 3 hours, 18 minutes Affected Services and Features:
Regions/Zones: Global Description: Multiple Google Cloud services experienced issues from 8 May 2022 23:24 to 9 May 2022 02:42 US/Pacific. From preliminary analysis, the root cause of the issue was network issues that caused tests to fail which led the configuration management system to treat the config as a bad config and prevented propagating configuration changes to Google Cloud infrastructure backend servers. Customer Impact:
|
| | 9 May 2022 | 02:49 PDT | The issue with Google App Engine, Google Cloud Composer, Google Cloud DNS, Google Cloud Networking, Google Compute Engine, Service Directory has been resolved for all affected users as of Monday, 2022-05-09 02:42 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 9 May 2022 | 02:27 PDT | Summary: Google Cloud Infrastructure is failing to push cloud configs Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Monday, 2022-05-09 04:00 US/Pacific. Diagnosis: Customers may experience the following: 1) External HTTP/S Load Balancing, Cloud CDN - config changes will be accepted but fail to propagate 2) Cloud Armor rules may also be affected 3) App Engine Flex customers may see apps fail to deploy 4) Traffic Director may see configs fail to propagate 5) VM instance group health checks & functionality like autoscaling/autohealing may be affected. Product Impact: Google App Engine Flex: Impact/Diagnosis: Users may observe error, DEADLINE_EXCEEDED. “Timed out waiting for the app infrastructure to become healthy”. This occurs when attempting to deploy major version updates. Workaround: None. Google Cloud Networking Impact/Diagnosis: Cloud customers modifying backend configuration for load balanced products, or health check configuration, may see their configuration accepted but not propagate, or fail after a timeout. Workaround: None Cloud Composer: Impact: Users are unable to create Composer 1 and Composer 2 "Private IP" environments. Workaround: No workaround is available for Composer 1. For Composer 2 “Private IP” the workaround is to create environments with Private Service Connect. Composer 2 “Public IP” environment creations should be successful. Workaround: None |
| | 9 May 2022 | 01:28 PDT | Summary: GICS is failing to push cloud configs Description: We are experiencing an issue with Google Cloud DNS, Service Directory beginning at Sunday, 2022-05-08 23:24 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2022-05-09 03:00 US/Pacific with current details. Diagnosis: Customer's might experience following:
Workaround: None |
- All times are US/Pacific