Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 29 March 2023 12:35

Incident End: 31 March 2023 03:00

Duration: 38 hours and 25 minutes

Affected Services and Features:

Google Cloud Networking

Regions/Zones: us-central1, us-east4, us-east5

Description:

Google Cloud Networking users experienced packet loss for 70 seconds and slightly elevated latency for 38 hours and 25 minutes for connectivity from us-central1 to us-east4 and us-east5.

From preliminary analysis, the root cause of the packet loss and subsequent latency was two independent instances of physical damage in North America, which left multiple regions operating on longer physical paths.

Service was restored by fixing the physical damage, followed by restoration of traffic on the impacted network. Subsequently, all network flow resumed full operation at 31 March 2023 06:30 US/Pacific.

Customer Impact:

Users experienced packet loss for 70 seconds on 29 March 2023 between 12:35 and 12:37. A small percentage of users experienced slightly elevated latency of up to 10 minutes until 31 March 2023 at 03:00 for connectivity between us-central1 to us-east4 and us-east5.

The issue with Google Cloud Networking has been resolved for all affected projects as of Friday, 2023-03-31 06:35 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: Most of the network flows have recovered to their original latency, with a small percentage of flows still seeing elevated latency.

These latencies are gradually recovering and the work is still ongoing to recover capacity as soon as possible and with the highest priority.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 10:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: The latencies are gradually recovering (P50 and P95 are now recovered, P99 is decreasing). The work is still ongoing to recover capacity as soon as possible by our onsite teams at the highest priority.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 06:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: The latencies are gradually recovering (P50 and P95 are now recovered, P99 is decreasing). The work is still ongoing to recover capacity as soon as possible by our onsite teams at the highest priority.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 05:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: While the engineering team continues the efforts to recover the capacity, we believe the issue with Google Cloud Identity-Aware Proxy, Google Cloud Networking, Identity and Access Management is no longer causing any impact due to packet loss.

However, we have identified an increase in latency for Google Cloud Networking affecting some regions, in particular us-central1, us-east4 and us-east5, due to physical damage resulting in a longer inter-regional path.

The latencies are gradually recovering but the work is still ongoing to recover capacity as soon as possible by our onsite teams at the highest priority, we will provide a full service restoration ETA once it is available.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 04:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: While the engineering team continues the efforts to recover the capacity, we believe the issue with Google Cloud Identity-Aware Proxy, Google Cloud Networking, Identity and Access Management is no longer causing any impact due to packet loss.

However, we have identified an increase in latency for Google Cloud Networking affecting some regions, in particular us-central1, us-east4 and us-east5, due to physical damage resulting in a longer inter-regional path.

Work is ongoing to recover capacity as soon as possible by our onsite teams at the highest priority, we will provide a full service restoration ETA once it is available.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 03:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: While the engineering team continues the efforts to recover the capacity, we believe the issue with Google Cloud Identity-Aware Proxy, Google Cloud Networking, Identity and Access Management is no longer causing any impact due to packet loss.

However, we have identified an increase in latency for Google Cloud Networking affecting some regions, in particular us-central1, us-east4 and us-east5, due to physical damage resulting in a longer inter-regional path.

Work is ongoing to recover capacity as soon as possible by our onsite teams at the highest priority, we will provide a full service restoration ETA once it is available.

We will continue to communicate any changes in status and will provide another update by Friday, 2023-03-31 00:00 US/Pacific with current details.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: We believe the issue with Google Cloud Identity-Aware Proxy, Google Cloud Networking, Identity and Access Management is no longer causing any impact due to packet loss.

However, we have identified an increase in latency for Google Cloud Networking affecting some regions, in particular us-central1, us-east4 and us-east5, due to physical damage resulting in a longer inter-regional path.

Work is ongoing to recover capacity as soon as possible by our onsite teams at the highest priority, we will provide a full service restoration ETA once it is available.

We will continue to communicate any changes in status and will provide another update by Thursday, 2023-03-30 21:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

Description: We believe the issue with Google Cloud Identity-Aware Proxy, Google Cloud Networking, Identity and Access Management is no longer causing any impact due to packet loss.

However, we have identified an increase in latency for Google Cloud Networking affecting some regions, in particular us-central1, us-east4 and us-east5, due to physical damage resulting in a longer inter-regional path.

We are continuing to work on recovering capacity as soon as possible by our onsite teams at the highest priority. We do not have an ETA for service restoration.

We will provide an update by Thursday, 2023-03-30 18:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers experienced packet loss for two events in the approximate time frames 12:30 - 12:40 and 13:40 - 13:50 US/Pacific on Wednesday, 2023-03-29.

Customers may experience slightly elevated latency between regions while traffic recovers.

Workaround: None at this time.