Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Operations, Google Kubernetes Engine, Cloud Logging, Cloud Data Fusion

GKE API seeing high latency and errors in us-central1

Incident began at 2022-06-14 03:03 and ended at 2022-06-14 06:22 (all times are US/Pacific).

Previously affected location(s)

Iowa (us-central1)

Date Time Description
14 Jun 2022 11:54 PDT

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 14 June 2022 03:03

Incident End: 14 June 2022 06:22

Duration: 3 hours, 19 minutes

Affected Services and Features:

Google Kubernetes Engine, Cloud Data Fusion, Cloud Logging

Regions/Zones: us-central1

Description:

Google Kubernetes Engine (GKE) experienced increased latency and errors from the GKE API in us-central1 for 3 hours and 19 minutes. From preliminary analysis, the root cause of the issue was an unexpected increase in backend traffic that was compounded by clients retrying failed requests. The issue was mitigated by scaling up the GKE control plane to handle the increased traffic.

Customer Impact:

  • Google Kubernetes Engine (GKE) customers may have experienced increased latency and errors for GKE API calls via gcloud and the Google Cloud Console UI. GKE cluster workloads continued to run; however customers may have been unable to perform operations such as confirm cluster status, trigger upgrades, view workloads, or create/delete clusters and node pools.

  • Cloud Data Fusion customers may have experienced instance creation and deletion failures in us-central1.

  • Cloud Logging customers may have experienced increased latency for suggested searches for GKE clusters in us-central1.
14 Jun 2022 06:26 PDT

The issue with Google Kubernetes Engine has been resolved for all affected users as of Tuesday, 2022-06-14 06:17 US/Pacific.

If customers are still experiencing issues, please raise the case via normal channels .

We thank you for your patience while we worked on resolving the issue.
14 Jun 2022 06:15 PDT

Summary: GKE API seeing high latency and errors in us-central1

Description: Mitigation work is still underway by our engineering team.

Customers who are able to use another region are advised to do so.

We will provide more information by Tuesday, 2022-06-14 06:35 US/Pacific.

Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful.

Workaround: None at this time
14 Jun 2022 05:34 PDT

Summary: GKE API seeing high latency and errors in us-central1

Description: Mitigation work is still underway by our engineering team.

Customers who are able to use another region are advised to do so.

We will provide more information by Tuesday, 2022-06-14 06:10 US/Pacific.

Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful.

Workaround: None at this time
14 Jun 2022 05:01 PDT

Summary: GKE API seeing high latency and errors in us-central1

Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2022-06-14 05:30 US/Pacific.}

We will provide more information by Tuesday, 2022-06-14 05:31 US/Pacific.

Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful.

Workaround: None at this time
14 Jun 2022 04:28 PDT

Summary: GKE API seeing high latency and errors in us-central1

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2022-06-14 05:00 US/Pacific.

Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful.

Workaround: None at this time
14 Jun 2022 04:00 PDT

Summary: GKE API seeing high latency and errors in us-central1

Description: We are experiencing an issue with Google Kubernetes Engine beginning at Tuesday, 2022-06-14 03:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2022-06-14 04:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful.

Workaround: None at this time