Service Health
-
-
-
-
Incident affecting Google Compute Engine
Google Engineer are investigating an issues with Cloud GCE.
Incident began at 2022-09-14 17:00 and ended at 2022-09-15 08:56 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
| Date | Time | Description | |
|---|---|---|---|
| | 15 Sep 2022 | 13:40 PDT | Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . (All Times US/Pacific) Incident Start: 14 September 2022 17:00 Incident End: 15 September 2022 08:56 Duration: 15 hours, 56 minutes Affected Services and Features: Google Compute Engine - Red Hat OS Regions/Zones: Global Description: Google Compute Engine customers running “yum” or “dnf” commands in Red Hat instances experienced 403 forbidden responses when attempting to install packages. The root cause was that the Red Hat Update infrastructure software update [1] combined with an invalid configuration caused a new build of the software to be unusable and that build was mistakenly released to prod via automation. [1] https://access.redhat.com/products/red-hat-update-infrastructure Customer Impact: 100 % of customers running “yum” in RHEL instances received a 403 - Forbidden response. All Red Hat instances attempting to get updates encountered failures for the duration of the outage. |
| | 15 Sep 2022 | 09:08 PDT | The issue with Google Compute Engine has been resolved for all affected users as of Thursday, 2022-09-15 08:56 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 15 Sep 2022 | 08:11 PDT | Summary: Google Engineer are investigating an issues with Cloud GCE. Description: We are experiencing an issue with Google Compute Engine beginning at Thursday, 2022-09-15 00:00 US/Pacific Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-09-15 09:35 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers using our RedHat licenses are currently unable to update/apply RedHat patches Workaround: none at present |
| | 15 Sep 2022 | 08:09 PDT | Summary: Google Engineer are investigating an issues with Cloud GCE. Description: We are experiencing an issue with Google Compute Engine beginning at Thursday, 2022-09-15 00:00 US/Pacific Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-09-15 09:35 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers using our RedHat licenses are currently unable to update/apply RedHat patches Workaround: none at present |
| | 15 Sep 2022 | 08:04 PDT | Summary: Google Engineer are investigating an issues with Cloud GCE. Description: We are experiencing an issue with Google Compute Engine {IF START TIME KNOWN, beginning at Thursday, 2022-09-15 00:00 US/Pacific}. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-09-15 09:35 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers using our RedHat licenses are currently unable to update/apply RedHat patches Workaround: none at present |
| | 15 Sep 2022 | 08:02 PDT | Summary: Google Engineer are investigating an issues with Cloud GCE. Description: We are experiencing an issue with Google Kubernetes Engine {IF START TIME KNOWN, beginning at Thursday, 2022-09-15 00:00 US/Pacific}. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-09-15 09:35 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers using our RedHat licenses are currently unable to update/apply RedHat patches Workaround: none at present |
- All times are US/Pacific