Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 17 October 2023 18:00

Incident End: 17 October 2023 20:58

Duration: 2 hours, 58 minutes

Affected Services and Features:

Google Cloud VMware Engine

Regions/Zones: australia-southeast1

Description:

Google Cloud VMware Engine experienced connectivity issues in australia-southeast1 for a duration of 2 hours, 58 minutes. From preliminary analysis, the root cause of the issue was the loss of multiple Virtual Private Cloud (VPC) peer circuits for the affected data center.

Customer Impact:

• Affected customers may have experienced connectivity issues for stretched clusters in the affected region.
• Hosts in vCenter may have been disconnected.
• Additionally, some customers may have experienced connectivity issues from their GCP and on-premises networks to their Private Clouds for the duration of the outage.

Additional details:

Google engineers brought up a backup transit link on the remaining circuit at 18:03 US/Pacific, which restored connectivity between stretch sites. Engineers then began migrating customer connections to the remaining circuit to restore access from on-premises sites. All Virtual Private Cloud (VPC) peer circuits were restored by 20:58 US/Pacific.

Some customers may have experienced residual impact beyond the resolution time due to the duration of time that their stretch sites were unreachable.

The issue with VMWare engine has been resolved for all affected users as of Tuesday, 2023-10-17 20:54 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Cloud VMWare engine is experiencing network link issues

Description: The impact for Stretch Clusters is now mitigated.

Our engineering team is continuing to work on mitigation for the rest of VMWare engine traffic in the region.

The mitigation is expected to complete by Tuesday, 2023-10-17 21:15 US/Pacific.

We will provide more information by Tuesday, 2023-10-17 21:30 US/Pacific.

Diagnosis: - Affected customers may experience issues with Apps responding.

• Hosts in vcenter may be disconnected.

Workaround: None at this time.

Summary: Google Cloud VMWare engine is experiencing network link issues

Description: The impact for Stretch Clusters is now mitigated.

Our engineering team is continuing to work on mitigation for the rest of VMWare engine traffic in the region.

The mitigation is expected to complete by Tuesday, 2023-10-17 20:15 US/Pacific.

We will provide more information by Tuesday, 2023-10-17 20:30 US/Pacific.

Diagnosis:

• Affected customers may experience issues with Apps responding.
• Hosts in vcenter may be disconnected.

Workaround: None at this time.

Summary: Google Cloud VMWare engine is experiencing network link issues

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-10-17 19:30 US/Pacific.

Diagnosis:

• Affected customers may experience issues with Apps responding.
• Hosts in vcenter may be disconnected.

Workaround: None at this time.