Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud Platform. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit cloud.google.com.

Google Cloud Networking Incident #19016

Cloud Networking issues in us-east1

Incident began at 2019-07-02 10:25 and ended at 2019-07-03 12:00 (all times are US/Pacific).

Date Time Description
Jul 03, 2019 12:00

The disruption with Google Cloud Networking and Load Balancing has been resolved as of Wednesday, 2019-07-03 07:35 US/Pacific.

The physical damage to the fiber bundles serving some network paths in us-east1 have been repaired, and we have removed the elective routing mitigations previously in place and returned to normal routing in the region.

We will perform an internal review of the incident and make appropriate improvements so that our customers continue to be able to operate reliably during incidents such as this.

The disruption with Google Cloud Networking and Load Balancing has been resolved as of Wednesday, 2019-07-03 07:35 US/Pacific.

The physical damage to the fiber bundles serving some network paths in us-east1 have been repaired, and we have removed the elective routing mitigations previously in place and returned to normal routing in the region.

We will perform an internal review of the incident and make appropriate improvements so that our customers continue to be able to operate reliably during incidents such as this.

Jul 02, 2019 16:05

The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours.

In the meantime, we are electively rerouting traffic to ensure that customers' services will continue to operate reliably until the affected fiber paths are repaired. Some customers may observe elevated latency during this period.

We will provide another status update either as the situation warrants or by Wednesday, 2019-07-03 12:00 US/Pacific tomorrow.

The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours.

In the meantime, we are electively rerouting traffic to ensure that customers' services will continue to operate reliably until the affected fiber paths are repaired. Some customers may observe elevated latency during this period.

We will provide another status update either as the situation warrants or by Wednesday, 2019-07-03 12:00 US/Pacific tomorrow.

Jul 02, 2019 14:31

The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1. We are electively rerouting some traffic to ensure that customers’ services will continue to operate reliably until the affected fiber paths are repaired. Some customers will observe elevated latency during this period.

We will provide another status update by Tuesday, 2019-07-02 16:30 US/Pacific with current details.

The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1. We are electively rerouting some traffic to ensure that customers’ services will continue to operate reliably until the affected fiber paths are repaired. Some customers will observe elevated latency during this period.

We will provide another status update by Tuesday, 2019-07-02 16:30 US/Pacific with current details.

Jul 02, 2019 12:39

Mitigation work is currently underway to address the issue with Google Cloud Networking and Load Balancing in us-east1. The rate of errors has decreased, however some users may still notice elevated latency, and our Engineering Team is continuing to work on it. We will provide another status update by Tuesday, 2019-07-02 14:30 US/Pacific with current details.

Mitigation work is currently underway to address the issue with Google Cloud Networking and Load Balancing in us-east1. The rate of errors has decreased, however some users may still notice elevated latency, and our Engineering Team is continuing to work on it. We will provide another status update by Tuesday, 2019-07-02 14:30 US/Pacific with current details.

Jul 02, 2019 11:35

Mitigation work is currently underway by our Engineering Team to address the issue with Google Cloud Networking and Load Balancing in us-east1. The rate of errors is decreasing, however some users may still notice elevated latency. We will provide another status update by Tuesday, 2019-07-02 12:30 US/Pacific with current details.

Mitigation work is currently underway by our Engineering Team to address the issue with Google Cloud Networking and Load Balancing in us-east1. The rate of errors is decreasing, however some users may still notice elevated latency. We will provide another status update by Tuesday, 2019-07-02 12:30 US/Pacific with current details.

Jul 02, 2019 10:56

We continue to experience an issue with Cloud Networking and Load balancing within us-east1. Issue is currently mitigated and we are working towards a resolution. Customers may still observe traffic through Global Load balancers being directed away from backends in us-east1 at this time. We will provide an update by Tuesday, 2019-07-02 11:29 US/Pacific with current details.

We continue to experience an issue with Cloud Networking and Load balancing within us-east1. Issue is currently mitigated and we are working towards a resolution. Customers may still observe traffic through Global Load balancers being directed away from backends in us-east1 at this time. We will provide an update by Tuesday, 2019-07-02 11:29 US/Pacific with current details.

Jul 02, 2019 10:25

We continue to experience an issue with Cloud Networking within us-east1. Issue is currently mitigated and we are working towards a resolution. Customers may still observe traffic through Global Load-balancers being directed away from back-ends in us-east1 at this time. We will provide an update by Tuesday, 2019-07-02 11:29 US/Pacific with current details.

We continue to experience an issue with Cloud Networking within us-east1. Issue is currently mitigated and we are working towards a resolution. Customers may still observe traffic through Global Load-balancers being directed away from back-ends in us-east1 at this time. We will provide an update by Tuesday, 2019-07-02 11:29 US/Pacific with current details.