Service Health
-
-
-
-
Incident affecting Google Cloud Infrastructure Components, Google Cloud Networking
We believe that the issue with Google Cloud infrastructure components has been mitigated. We are continuing to actively monitor the situation.
Incident began at 2020-03-26 08:18 and ended at 2020-03-26 09:21 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 26 Mar 2020 | 13:10 PDT | Some of our users experienced a service disruption today, as a result of a significant router failure at 08:18am Pacific in one of our data centers in Atlanta, causing network congestion. As a result, Google services running in that data center were directly impacted and were unavailable until our engineers rerouted the traffic and moved those services to alternate facilities. Users in the South Eastern US may also have seen temporary difficulties in accessing a wider range of Google services due to the network congestion. The majority of directly impacted Google services were moved to alternative data centers by 08:50am Pacific, and networking impact was mitigated by 09:21am Pacific, with some services taking longer to recover. The root cause of the problem was a failure of networking devices at our Atlanta datacenter site. We are working on mitigating the issue and taking steps to avoid a recurrence. |
| | 26 Mar 2020 | 10:38 PDT | The issue with Google Cloud infrastructure components has been resolved for all affected users as of 09:21. Total time of impact was 08:18 to 09:21 US/Pacific, with the most severe impact at the start of the issue, tapering off as services routed traffic away from Atlanta. The impact of this incident was concentrated in a region that is not a main GCP region and therefore there was no impact to services based on Google Compute Engine. Services that may have been impacted include External HTTP/S Load balancing requests and API requests that may have been served near the Atlanta metro. The root cause was a set of router failures in Atlanta, which affected traffic routed through that region. |
| | 26 Mar 2020 | 10:17 PDT | Description: We believe the issue with Google Cloud infrastructure components is mitigated, and we are continuing to actively monitor the situation. We will provide an update by Thursday, 2020-03-26 12:00 US/Pacific with current details. Diagnosis: None at this time Workaround: None at this time |
| | 26 Mar 2020 | 09:38 PDT | Description: We believe the issue with Google Cloud infrastructure components is mitigated, and we are continuing to actively monitor the situation. We will provide an update by Thursday, 2020-03-26 10:15 US/Pacific with current details. Diagnosis: None at this time Workaround: None at this time |
| | 26 Mar 2020 | 09:32 PDT | Description: We believe the issue with Google Cloud infrastructure components is partially resolved and recovery is beginning. We do not have an ETA for full resolution at this point. We will provide an update by Thursday, 2020-03-26 10:00 US/Pacific with current details. Diagnosis: None at this time Workaround: None at this time |
| | 26 Mar 2020 | 09:08 PDT | Description: We are experiencing an issue with Google Cloud infrastructure components. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2020-03-26 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None at this time Workaround: None at this time |
- All times are US/Pacific