Service Health
-
-
-
-
Incident affecting Google Kubernetes Engine
europe-central2, europe-west3: Elevated error rates for GKE control plane
Incident began at 2021-09-09 12:06 and ended at 2021-09-09 12:54 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 9 Sep 2021 | 15:25 PDT | We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 09 September 2021 10:46 Incident End: 09 September 2021 12:44 Duration: 1 hour, 58 minutes Affected Services and Features:
Regions/Zones: europe-west3 , europe-central2 Description: Google Kubernetes Engine (GKE) experienced elevated error rates intermittently in europe-west3 and europe-central2 for a duration of 1 hour, 58 minutes. From preliminary analysis, the issue was caused by an ongoing rollout in the backend authentication service. The backend authentication service was rolled back, ending impact at 12:44. Customer Impact:
Additional details: A retry of the failed operation would have completed successfully. |
| | 9 Sep 2021 | 12:54 PDT | The issue with Google Kubernetes Engine has been resolved for all affected projects as of Thursday, 2021-09-09 12:53 US/Pacific. There is no further impact to our customers. We thank you for your patience while we worked on resolving the issue. |
| | 9 Sep 2021 | 12:48 PDT | Summary: europe-central2, europe-west3: Elevated error rates for GKE control plane Description: Our engineering team is currently working on the mitigation of the issue. The affected locations have reduced and updated for the incident post. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2021-09-09 14:00 US/Pacific. Diagnosis: Customers may experience issue with cluster availability and errors with CreateToken. Service account may intermittently succeed or fail. Workaround: Customers who experience failure can retry. |
| | 9 Sep 2021 | 12:29 PDT | Summary: europe-central2, europe-west3, europe-west4, us-east4, asia-northeast1: Elevated error rates for GKE control plane Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2021-09-09 14:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may experience issue with cluster availability and errors with CreateToken. Workaround: None at this time. |
| | 9 Sep 2021 | 12:18 PDT | Summary: europe-central2, europe-west3, europe-west4, us-east4, asia-northeast1: Elevated error rates for GKE control plane Description: We are investigating a potential issue with Google Kubernetes Engine. We will provide more information by Thursday, 2021-09-09 12:35 US/Pacific. Diagnosis: We are currently investigating the impact of this incident. Workaround: None at this time. |
| | 9 Sep 2021 | 12:06 PDT | Summary: europe-west3, europe-west4, us-east4, asia-northeast1: Elevated error rates for GKE control plane Description: We are investigating a potential issue with Google Kubernetes Engine. We will provide more information by Thursday, 2021-09-09 12:35 US/Pacific. Diagnosis: We are currently investigating the impact of this incident. Workaround: None at this time. |
- All times are US/Pacific