Service Health

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 October 2022 03:00

Incident End: 02 November 2022 20:00

Duration: 9 days, 17 hours

Affected Services and Features:

Virtual Private Cloud

Regions/Zones: global

Description:

Virtual Private Cloud (VPC) experienced connectivity issues between instances and/or with on-premises machines. In some cases, data plane components in affected regions did not get dynamic routes from peered networks.

From preliminary analysis, the root cause of the issue is a rollout to virtual network controllers. The rollout started on 24 October 2022 to a small subset of controllers and gradually scaled over time to other controllers.

The issue was mitigated on 02 November 2022 by rolling back.

Customer Impact:

Customers were unable to access their Cloud Volume Service (CVS) storage across GCP Regions. Customers may have faced issues with traffic routing In some cases where dynamic routes were not propagated appropriately on peered networks across regions.

We experienced an issue with Virtual Private Cloud (VPC) beginning at Wednesday, 2022-11-02 02:46 US/Pacific.

The issue has been resolved for all affected users as of Wednesday, 2022-11-02 20:14 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: Mitigation work is still underway by our engineering team.

Mitigation has been fully applied in: europe-west1-b, europe-west1-c, europe-west1-d, us-west1, us-west2.

The engineering team has started a rollout of the mitigation globally.

We will provide more information by Wednesday, 2022-11-02 20:30 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: Mitigation work is still underway by our engineering team.

Mitigation has been fully applied in: europe-west1-b, europe-west1-c, europe-west1-d, us-west1, us-west2.

The engineering team has started a rollout of the mitigation globally.

The mitigation is expected to complete by Wednesday, 2022-11-02 20:00 US/Pacific.}

We will provide more information by Wednesday, 2022-11-02 20:00 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: Mitigation work is still underway by our engineering team.

Mitigation has been fully applied in: europe-west1-b, europe-west1-c, europe-west1-d, us-west1, us-west2.

The engineering team has started a rollout of the mitigation globally.

The mitigation is expected to complete by Wednesday, 2022-11-02 18:00 US/Pacific.}

We will provide more information by Wednesday, 2022-11-02 18:00 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: Mitigation work is still underway by our engineering team.

Mitigation has been fully applied in: europe-west1-b, europe-west1-c, europe-west1-d, us-west1, us-west2.

The engineering team has started a rollout of the mitigation globally.

The mitigation is expected to complete by Wednesday, 2022-11-02 16:00 US/Pacific.}

We will provide more information by Wednesday, 2022-11-02 16:00 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: We've received a report of an issue with Virtual Private Cloud (VPC).

Mitigation has been fully applied in: europe-west1-b, europe-west1-c, europe-west1-d, us-west1 us-west2.

The engineering team has started a rollout of the mitigation globally.

We do not have an ETA for rolling out mitigation globally at this point.

We will provide more information by Wednesday, 2022-11-02 14:43 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: We've received a report of an issue with Virtual Private Cloud (VPC).

Mitigation work has been fully applied in europe-west-1-B, europe-west-1-C, and europe-west-1-D and it continues to be rolled out to other regions by the engineering team.

We do not have an ETA for rolling out mitigation globally at this point.

We will provide more information by Wednesday, 2022-11-02 07:43 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: We've received a report of an issue with Virtual Private Cloud (VPC).

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-11-02 06:43 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: We've received a report of an issue with Virtual Private Cloud (VPC).

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-11-02 05:43 US/Pacific.

Diagnosis: Some users are experiencing connectivity issues between VMs or with onprem. Some data plane components in affected regions are in some cases not getting dynamic routes from peered networks.

Workaround: None at this time.

Summary: Some VPCs are missing dynamic routes from peered networks

Description: We've received a report of an issue with Virtual Private Cloud (VPC).

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-11-02 04:25 US/Pacific.

Diagnosis: Some customers are unable to access their Cloud Volume Service (CVS) storage across GCP Regions

Workaround: None at this time.