Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking

Cloud Router creations are failing in southamerica-east1

Incident began at 2021-08-16 01:19 and ended at 2021-08-17 08:55 (all times are US/Pacific).

Date Time Description
17 Aug 2021 14:06 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 16 August 2021 01:19

Incident End: 17 August 2021 08:55

Duration: 1 day, 7 hours, 36 minutes

Affected Services and Features: Cloud Router

Regions/Zones: southamerica-east1

Description:

Google Cloud Router experienced failures creating or modifying routers in southamerica-east1 for a duration of 1 day, 7 hours and 36 minutes. New Cloud router creations or modifications to existing Cloud Routers would have been impacted. From preliminary analysis, the root cause of the issue is related to a rollout within the virtual routing control plane for the region, which had a configuration mismatch as part of an ongoing migration.

Customer Impact:

  • Customers were unable to create or modify existing Cloud Routers from 16 August 2021 01:19 to 17 August 2021 08:34.
  • Router status requests [1] which rely on the getRouterStatus [2] API may have also returned errors during the incident
  • All existing routers that had completed programming continued to work throughout the entire duration.
  • Dynamic routing functionality [3] using BGP routes continued to work as expected.

Additional details:

  • In order to prevent a recurrence, we are fixing outdated automations used during migrations
  • In order to reduce detection delays, we are improving our control plane probers to ensure we are alerted promptly

Reference(s):
[1] https://cloud.google.com/network-connectivity/docs/router/how-to/viewing-router-details#viewing-router-status
[2] https://cloud.google.com/compute/docs/reference/rest/v1/routers/getRouterStatus
[3] https://cloud.google.com/vpc/docs/routes#dynamic_routes

17 Aug 2021 10:14 PDT

The issue with Cloud Router has been resolved for all affected projects as of Tuesday, 2021-08-17 10:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

17 Aug 2021 09:08 PDT

Summary: Cloud Router creations are failing in southamerica-east1

Description: We believe the issue with Cloud Router is partially resolved. Full resolution is expected to complete by Tuesday, 2021-08-17 10:15 US/Pacific. We will provide an update by Tuesday, 2021-08-17 10:15 US/Pacific with current details.

Diagnosis: New Cloud Router creations should be failing in southamerica-east1. Status request operations may also fail.

Workaround: Avoid making changes to existing router within southamerica-east1.

17 Aug 2021 08:17 PDT

Summary: Cloud Router creations are failing in southamerica-east1

Description: We are experiencing an issue with Cloud Router pertaining to creations within southamerica-east1. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-08-17 09:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: New Cloud Router creations should be failing in southamerica-east1. Status request operations may also fail

Workaround: Avoid making changes to existing router within southamerica-east1.