Service Health
-
-
-
-
Incident affecting Google Cloud Infrastructure Components, Cloud Developer Tools, Cloud Endpoints, Eventarc, Google App Engine, Google Cloud Bigtable, Google Cloud Console, Google Cloud Networking, Operations, Container Registry, Cloud Spanner, Google Cloud SQL, Cloud Load Balancing
We are experiencing Networking issues
Incident began at 2021-11-12 00:30 and ended at 2021-11-12 02:14 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 12 Nov 2021 | 16:43 PST | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 12 November 2021 00:30 Incident End: 12 November 2021 02:14 Duration: 1 hour 44 minutes Affected Services and Features:
Regions/Zones: Europe Description: Google’s Front End load balancing service experienced failures resulting in impact to several downstream Google Cloud services in Europe. From preliminary analysis, the root cause of the issue was caused by a new infrastructure feature triggering a latent issue within internal network load balancer code. Customer Impact:
Additional details: The error was caught within 4 minutes by automated safety systems, and further spread was slowed at this point. The issue was fully mitigated approximately 1hr 44m later, when our engineering team completed a rollout to disable the vulnerable code path. The issue will be fully prevented going forward via a root cause fix, which will complete rollout by 12 November 2021 21:00 US/Pacific. |
| | 12 Nov 2021 | 03:44 PST | The issue with Google Cloud infrastructure components has been resolved for all affected products as of Friday, 2021-11-12 03:38 US/Pacific. If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We thank you for your patience while we worked on resolving the issue. |
| | 12 Nov 2021 | 03:38 PST | Summary: We are experiencing Networking issues Description: We believe the issue with Google Cloud infrastructure components is partially resolved with the exception of the Google App Engine that is also recovering. We will provide an update by Friday, 2021-11-12 04:10 US/Pacific with current details. Diagnosis: Customers might face connectivity issues. Workaround: None |
| | 12 Nov 2021 | 02:57 PST | Summary: We are experiencing Networking issues Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point but the services should be improving at this point. The issue is affecting:
Diagnosis: Customers might face connectivity issues. Workaround: None |
| | 12 Nov 2021 | 02:36 PST | Summary: We are experiencing Networking issues Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. The issue is affecting,
Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-11-12 03:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers might face connectivity issues. Workaround: None |
| | 12 Nov 2021 | 02:26 PST | Summary: We are experiencing Networking issues Description: We are experiencing an issue with Google Cloud infrastructure components beginning at Friday, 2021-11-12 00:36:24 PST US/Pacific. The issue is affecting,
Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-11-12 02:45 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers might face connectivity issues. Workaround: None |
| | 12 Nov 2021 | 02:13 PST | Summary: We are experiencing Networking issues Description: We are experiencing an issue with Google Cloud infrastructure components beginning at Friday, 2021-11-12 00:36:24 PST US/Pacific. The issue is affecting,
Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-11-12 02:45 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None Workaround: None |
| | 12 Nov 2021 | 01:42 PST | Summary: We are experiencing Networking issues Description: We are experiencing an issue with Google Cloud infrastructure components beginning at Friday, 2021-11-12 00:36:24 PST US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-11-12 02:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None Workaround: None |
- All times are US/Pacific