Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 16 March 2023 12:50

Incident End: 19 March, 2023 09:24

Duration: 2 days, 20 hours, 34 minutes

Affected Services and Features:

Google Cloud Networking - Google Cloud Load Balancing

Regions/Zones:

• us-east4
• us-east1
• us-east5

Description:

Google Front End (GFE) tasks in us-east4 were overloaded causing some traffic to be redirected to other regions including us-east1 and us-east5, which in turn caused higher than normal latency and errors for a duration of 2 days, 20 hours, 34 minutes.

From preliminary analysis, the root cause was due to an unexpected increase in traffic. Our engineering team is working to increase capacity in the us-east region to prevent a recurrence which is expected to complete by 21 March 2023.

Customer Impact:

• Google Cloud Load Balancer traffic in affected regions experienced higher latency and possible elevated error rates during peak usage over the last few days (~05:00 to ~19:00 US/Pacific).

Additional details:

• There were no workarounds for customers who were required to host in us-east regions during the incident.
• For customers who could move their backends to other regions (e.g. into us-west, europe and asia) this could have been a valid mitigation for them.

The issue with Cloud Load Balancing has been resolved for all affected users as of Monday, 2023-03-20 04:46 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2023-03-20 06:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2023-03-20 04:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2023-03-20 03:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-03-19 22:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-03-19 21:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Our engineering team continues to work on the mitigation of the issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-03-19 18:30 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-03-19 18:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-03-19 15:00 US/Pacific.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Google Engineers are investigating high latency and elevated errors in US-East.

Description: We are experiencing an issue with Cloud Load Balancing.

Our engineering team continues to investigate the issue. Engineers are working on mitigation plan. Customers are advised to move workloads out of US-East to avoid impact

We will provide an update by Sunday, 2023-03-19 12:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions

Description: We are experiencing an issue with Cloud Load Balancing.

Our engineering team continues to investigate the issue. Engineers are working on mitigation plan. Customers can move workloads out of US-East to avoid impact

We will provide an update by Sunday, 2023-03-19 12:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east. Customer moving workloads to us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions

Description: We are experiencing an issue with Cloud Load Balancing.

Our engineering team continues to investigate the issue.

We will provide an update by Sunday, 2023-03-19 09:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east - us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions

Description: We are experiencing an issue with Cloud Load Balancing.

Our engineering team continues to investigate the issue.

We will provide an update by Sunday, 2023-03-19 07:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east - us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.

Summary: Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions

Description: We are experiencing an issue with Cloud Load Balancing.

Our engineering team continues to investigate the issue.

We will provide an update by Sunday, 2023-03-19 07:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)

Workaround: Customers who are able to move their backends to regions away from us-east - us-west, europe and asia - may see improvements. No workarounds at this time for customers who are required to host in us-east regions.