Incident affecting VMWare engine, Google Compute Engine
vSphere vCenter and NSX may not have been available for a short period of time in us-central1
Incident began at 2022-06-23 00:13 and ended at 2022-06-23 06:36 (all times are US/Pacific).
Previously affected location(s)
| ||23 Jun 2022||14:57 PDT|| |
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
Incident Start: 23 June 2022 00:13 PT
Incident End: 23 June 2022 06:36 PT
Duration: 6 hours, 23 minutes
Affected Services and Features:
Google Cloud customers using VMWare Engine experienced connectivity issues while using vSphere, vCenter and NSX in us-central1a zone for a duration of 6 hours, 23 minutes.
From preliminary analysis, the issue was caused due to a network switch-over not working as expected during a planned maintenance window of ToR switches. The issue was mitigated by performing a file system recovery and restart of related services on the resources hosted by Google Cloud VMware Engine.
Google engineers have paused the maintenance upgrade and also all redundant devices have been brought back to operational state. We are also working with the impacted customers to check the state of remaining workload VMs and shall assist in recovery if required.
| ||23 Jun 2022||06:36 PDT|| |
The issue with VMWare engine has been resolved for all affected projects as of Thursday, 2022-06-23 06:36 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||23 Jun 2022||04:02 PDT|| |
Summary: vSphere vCenter and NSX may not have been available for a short period of time in us-central1
Description: We are experiencing an issue with VMWare engine beginning at Thursday, 2022-06-23 07:13 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Thursday, 2022-06-23 09:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: vSphere vCenter and NSX may not have been available for a short period of time in us-central1, vCenter VMs has been recovered. Some of the Workload VMs might be still impacted.
Workaround: None at this time.