Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud Platform. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit cloud.google.com.

Google Cloud Networking Incident #20005

We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d

Incident began at 2020-06-29 08:20 and ended at 2020-06-29 13:06 (all times are US/Pacific).

Date Time Description
Jun 29, 2020 13:06

The issue with Cloud Networking and Persistent Disk has been resolved for the majority of affected projects as of Monday, 2020-06-29 10:20 US/Pacific, and we expect full mitigation to occur for remaining projects within the hour.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.

We will publish analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we're working on resolving the issue.

The issue with Cloud Networking and Persistent Disk has been resolved for the majority of affected projects as of Monday, 2020-06-29 10:20 US/Pacific, and we expect full mitigation to occur for remaining projects within the hour.

If you have questions or feel that you may be impacted, please open a case with the Support Team and we will work with you until the issue is resolved.

We will publish analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we're working on resolving the issue.

Jun 29, 2020 12:17

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 13:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 13:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 11:45

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 12:20 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 12:20 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 11:00

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 11:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 11:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 10:31

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 10:00

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Services in us-east1-c are still being restored. No ETA as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 10:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been restored. Services in us-east1-c are still being restored. No ETA as of now.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services.

We will provide an update by Monday, 2020-06-29 10:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 09:22

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

We expect services in us-east1-d to be recovering within the next 30 minutes, there is no eta for service recovery for us-east1-c yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services..

We will provide an update by Monday, 2020-06-29 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

We expect services in us-east1-d to be recovering within the next 30 minutes, there is no eta for service recovery for us-east1-c yet.

Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services..

We will provide an update by Monday, 2020-06-29 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

Jun 29, 2020 08:48

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning at Monday,2020-06-29 08:15 US/Pacific, affecting multiple Google Cloud Services.

Symptoms: Connections to and from VMs in us-east1-c and us-east1-d may fail.

Our engineering team continues to investigate the issue..

We will provide an update by Monday, 2020-06-29 10:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: VM networking in us-east1-c and us-east1-d are failing.

Workaround: None at this time.

Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning at Monday,2020-06-29 08:15 US/Pacific, affecting multiple Google Cloud Services.

Symptoms: Connections to and from VMs in us-east1-c and us-east1-d may fail.

Our engineering team continues to investigate the issue..

We will provide an update by Monday, 2020-06-29 10:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: VM networking in us-east1-c and us-east1-d are failing.

Workaround: None at this time.