Service Health

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 16 August 2021 01:19

Incident End: 17 August 2021 08:55

Duration: 1 day, 7 hours, 36 minutes

Affected Services and Features: Cloud Router

Regions/Zones: southamerica-east1

Description:

Google Cloud Router experienced failures creating or modifying routers in southamerica-east1 for a duration of 1 day, 7 hours and 36 minutes. New Cloud router creations or modifications to existing Cloud Routers would have been impacted. From preliminary analysis, the root cause of the issue is related to a rollout within the virtual routing control plane for the region, which had a configuration mismatch as part of an ongoing migration.

Customer Impact:

• Customers were unable to create or modify existing Cloud Routers from 16 August 2021 01:19 to 17 August 2021 08:34.
• Router status requests [1] which rely on the getRouterStatus [2] API may have also returned errors during the incident
• All existing routers that had completed programming continued to work throughout the entire duration.
• Dynamic routing functionality [3] using BGP routes continued to work as expected.

Additional details:

• In order to prevent a recurrence, we are fixing outdated automations used during migrations
• In order to reduce detection delays, we are improving our control plane probers to ensure we are alerted promptly

Reference(s):
[1] https://cloud.google.com/network-connectivity/docs/router/how-to/viewing-router-details#viewing-router-status
[2] https://cloud.google.com/compute/docs/reference/rest/v1/routers/getRouterStatus
[3] https://cloud.google.com/vpc/docs/routes#dynamic_routes

The issue with Cloud Router has been resolved for all affected projects as of Tuesday, 2021-08-17 10:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Router creations are failing in southamerica-east1

Description: We believe the issue with Cloud Router is partially resolved. Full resolution is expected to complete by Tuesday, 2021-08-17 10:15 US/Pacific. We will provide an update by Tuesday, 2021-08-17 10:15 US/Pacific with current details.

Diagnosis: New Cloud Router creations should be failing in southamerica-east1. Status request operations may also fail.

Workaround: Avoid making changes to existing router within southamerica-east1.

Summary: Cloud Router creations are failing in southamerica-east1

Description: We are experiencing an issue with Cloud Router pertaining to creations within southamerica-east1. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-08-17 09:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: New Cloud Router creations should be failing in southamerica-east1. Status request operations may also fail

Workaround: Avoid making changes to existing router within southamerica-east1.