INCIDENT REPORT

Summary:

On Tuesday, 7 June 2022, Google Cloud Networking experienced increased packet loss for egress traffic from Google to users and customers in the Middle East and elevated latency between our Europe and Asia Regions for 3 hours and 12 minutes, affecting several Cloud Networking products.

To our customers that were impacted during this outage, we sincerely apologize. We are conducting an internal investigation and are taking steps to improve our service.

Root Cause:

The outage was triggered by two simultaneous fiber cuts within our Middle East network. This affected the end-to-end path for several submarine cables, reducing capacity for many telecom and technology companies, including Google.

The reduced capacity increased latency and caused packet loss between Google regions in Europe and regions in the Middle East and the rest of Asia.

Remediation and Prevention:

Google Engineers were alerted to the network bandwidth drop on Tuesday, 7 June 2022 at 06:00 US/Pacific. Google Engineers immediately started to redirect Internet egress demand away from the region. At 07:02, Europe-Asia traffic was redirected across backup circuits via America, trading increased latency for lower packet loss. Fiber engineers restored interim capacity by 08:26, ending packet loss between Europe and the Middle East. Once final stability was confirmed, Europe-Asia traffic was moved back to the direct path at 09:02.

Google is committed to improving our service in the future and will be completing the following actions:

• We are increasing the resilience of terrestrial network paths to provide more resilience to dual failures.
• We are continuing with our planned programme of additional investments in the Europe-Middle East network to offer alternative network paths.
• We are improving the effectiveness of tooling to reduce Internet egress demand during capacity shortfalls.

Detailed Description of Impact:

On 07 June 2022 05:50 from 05:50 to 09:03 US/Pacific:

Cloud Load Balancing

Customers using Cloud Load Balancing would have experienced increased packet loss up to 60% for egress traffic from Google to the Middle East and increased network latency between our Europe and Asia regions.

Cloud Interconnect

Cloud Interconnect customers peering in the Middle East region would have experienced packet loss up to 80% during the incident.

Virtual Private Cloud (VPC)

Customers using Virtual Private Cloud requiring cross-regional traffic between the following regions would have experienced packet loss of up to 50% between 05:47 and 07:03, then elevated network latency until 09:03:

• asia-east1 to europe-southwest1, europe-west6, and europe-west8;
• asia-east2 to all regions in Europe;
• asia-south1 and asia-south2 to all regions in Europe, all northamerica-northeast and northamerica-southeast, and all us-east regions;
• asia-southeast1 and asia-southeast2 to all regions in Europe;
• australia-southeast2 to europe-west8.

Other Services

While not specifically called out, all Google Cloud Services in the Middle East and our Europe and Asia regions could have been affected by the packet loss for egress traffic from Google to the Middle East, as well as increased latency between our Europe and Asia regions. We have listed the major services that our telemetry identified as causing customer impact.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. We will provide a full Incident Report with additional details.

(All Times US/Pacific)

Incident Start: 07 June 2022 05:50

Incident End: 07 June 2022 09:02

Duration: 3 hours, 12 minutes

Affected Services and Features:

Google Cloud Networking Cloud NAT Hybrid Connectivity Virtual Private Cloud (VPC) Cloud Interconnect Google Kubernetes Engine (GKE)

Regions/Zones: Packet Loss (Global to Middle East Internet Users), Latency (Between Asia and Europe regions)

Description:

Google Cloud Networking experienced increased packet loss for egress traffic from Google to the Middle East, and elevated latency between our Europe and Asia Regions as a result, for 3 hours and 12 minutes, affecting several related products including Cloud NAT, Hybrid Connectivity and Virtual Private Cloud (VPC). From preliminary analysis, the root cause of the issue was a capacity shortage following two simultaneous fiber-cuts.

Customer Impact:

Affected customers would have experienced increased packet loss and increased latency for the affected metros which would have caused delays and timeouts for the following services :

• Cloud Load Balancing
• Cloud Interconnect
• Virtual Private Cloud (VPC)
• Cloud NAT
• Google Kubernetes Engine (GKE)

The issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) has been resolved for all affected users as of Tuesday, 2022-06-07 09:02 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Packet loss observed from Internet in Middle East to Google

Description: We believe the issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) is partially resolved. Packet loss should be mitigated, but some traffic continues to have elevated latency.

Full resolution is expected to complete by Tuesday, 2022-06-07 09:30 US/Pacific.

We will provide an update by Tuesday, 2022-06-07 09:30 US/Pacific with current details.

Diagnosis: Increased packet loss for traffic traversing the affected metros.

Workaround: None at this time.

Summary: Packet loss observed from Internet in Middle East to Google

Description: Mitigation work is currently underway by our engineering team to redirect traffic and increase capacity.

The mitigation is expected to complete around Tuesday, 2022-06-07 09:00 US/Pacific.

We will provide more information by Tuesday, 2022-06-07 09:30 US/Pacific.

Diagnosis: Increased packet loss for traffic traversing the affected metros.

Workaround: None at this time.

Summary: Packet loss observed from Internet in Middle East to Google

Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific.

There is a latency in connectivity between Europe and Asia.

Mitigation work is still underway by our engineering team.

We will provide more information by Tuesday, 2022-06-07 09:15 US/Pacific.

Diagnosis: Increased packet loss for traffic traversing the affected metros.

Workaround: None.

Summary: Major packet loss observed from Internet in Middle East to Google

Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2022-06-07 08:00 US/Pacific with current details.

Diagnosis: Increased packet loss for traffic traversing the affected metros.

Workaround: None.

Summary: Major packet loss observed between multiple regions - Europe and Asia

Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Tuesday, 2022-06-07 06:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2022-06-07 08:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Increased packet loss for traffic traversing the affected metros.

Workaround: None.