Service Health
Incident affecting Google Compute Engine, VMWare engine
Private Cloud Certificate’s renewal (DigiCert)
Incident began at 2024-07-30 02:36 and ended at 2024-07-30 12:44 (all times are US/Pacific).
Previously affected location(s)
Tokyo (asia-northeast1)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Sydney (australia-southeast1)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)Northern Virginia (us-east4)Los Angeles (us-west2)
Date | Time | Description | |
---|---|---|---|
| 30 Jul 2024 | 12:44 PDT | The issue with VMWare engine has been resolved for all affected projects as of Tuesday, 2024-07-30 12:30 US/Pacific. Google engineers have completed renewal of all certificates. We thank you for your patience while we worked on resolving the issue. Certificate renewals were done on NSX, V-Center and HCX virtual machines. 3rd party products might still see failures due to an SSL thumbprint mismatch. Customers should check third party applications to ensure backups, replication and other third party tools are working properly with vCenter. If there are any issues noted, please reach out to Google Cloud Support. |
| 30 Jul 2024 | 11:54 PDT | Summary: Private Cloud Certificate’s renewal (DigiCert) Description: DigiCert is a digital certificate authority that issues certificates used in your Google Cloud VMware Engine Private Cloud. DigiCert informed us of a recent incident in which they found a problem with their Domain Control Validation (DCV). As a result, by July 30, 2024, 19:30 UTC (12:30 US/Pacific), all certificates must be revoked without exception. Google engineers are actively working to renew the certificates on your private cloud before 19:30 UTC (12:30 US/Pacific). Certificate renewals are being done on NSX, V-Center and HCX virtual machines and will not have any impact on accessing any workload VMs. During the certificate renewal (which typically takes 30 mins), currently active backup & replication tasks and management tasks such as node addition/removal may fail. These tasks and new migration can't be started during this period. After the certificates are renewed, 3rd party products might still see failures due to thumbprint mismatch. Customers should check for third party applications. If there are any issues noted, please reach out to Google Cloud Support. We will provide an update by Tuesday, 2024-07-30 12:30 US/Pacific with current details. Diagnosis: No current impact to the customers. Workaround: None at this time. |
| 30 Jul 2024 | 10:39 PDT | Summary: Private Cloud Certificate’s renewal (DigiCert) Description: DigiCert is a digital certificate authority that issues certificates used in your Google Cloud VMware Engine Private Cloud. DigiCert informed us of a recent incident in which they found a problem with their Domain Control Validation (DCV). As a result, by July 30, 2024, 19:30 UTC (12:30 US/Pacific), all certificates must be revoked without exception. Google engineers are actively working to renew the certificates on your private cloud before 19:30 UTC (12:30 US/Pacific). Certificate renewals are being done on NSX, V-Center and HCX virtual machines and will not have any impact on accessing any workload VMs. During the certificate renewal (which typically takes 30 mins), currently active backup & replication tasks and management tasks such as node addition/removal may fail. These tasks and new migration can't be started during this period. We will provide an update by Tuesday, 2024-07-30 12:30 US/Pacific with current details. Diagnosis: No current impact to the customers. Workaround: None at this time. |
| 30 Jul 2024 | 02:36 PDT | Summary: Private Cloud Certificate’s renewal (DigiCert) Description: DigiCert is a digital certificate authority that issues certificates used in your Google Cloud VMware Engine Private Cloud. DigiCert informed us of a recent incident in which they found a problem with their Domain Control Validation (DCV). As a result, by July 30, 2024, 19:30 UTC (12:30 US/Pacific), all certificates must be revoked without exception. Google engineers will renew the certificates on your private cloud before 19:30 UTC (12:30 US/Pacific). Certificate renewal will be done on NSX, V-Center and HCX virtual machines and will not have any impact on accessing any workload VMs. During the certificate renewal (which typically takes 30 mins), currently active backup & replication tasks and management tasks such as node addition/removal may fail. These tasks and new migration can't be started during this period. We will provide an update by Tuesday, 2024-07-30 10:30 US/Pacific with current details. Diagnosis: No current impact to the customers. Workaround: None at this time. |
- All times are US/Pacific