Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Bigtable, Google Cloud Tasks

Cloud Bigtable customers experiencing errors and increased latency in us-central1

Incident began at 2023-12-04 19:50 and ended at 2023-12-06 03:55 (all times are US/Pacific).

Previously affected location(s)

Iowa (us-central1)

Date Time Description
7 Dec 2023 12:59 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 4 December 2023 19:50

Incident End: 6 December 2023 03:55

Duration:

  • Partial mitigation: 15 hours
  • Full mitigation: 1 day, 8 hours, 5 minutes

Affected Services and Features:

Google Cloud Bigtable

Regions/Zones: us-central1

Description:

Approximately 0.06% of Google Cloud Bigtable dataplane requests experienced increased latency and errors in the us-central1 region. A partial mitigation was applied 15 hours from the start of the incident and the incident was fully mitigated after 1 day, 8 hours and 5 minutes. The root cause of the issue was an isolation failure in the machines that were running Cloud Bigtable servers, this manifested primarily as a low rate of network packet loss. The change that triggered the isolation issue has been identified and rolled back.

Customer Impact:

Impacted Bigtable customers experienced increased latency and low level of connection errors in the us-central1 region.

Additional details:

The issue was first detected by automated monitoring on 4 December 2023 at 23:35 and engineers immediately started an investigation.

As part of the investigation, on 5 December 2023 at 14:40, engineers redirected some traffic away from a subset of the affected machines which partially mitigated impact.

Engineers subsequently identified the trigger, and mitigated the issue on 6 December 2023 at 03:55, which fully resolved the latency. Additionally, engineers have put in place controls to prevent recurrence.

6 Dec 2023 08:23 PST

The issue with Google Cloud Bigtable, Google Cloud Tasks has been resolved for all affected users as of Wednesday, 2023-12-06 08:23 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

6 Dec 2023 03:00 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable, Google Cloud Tasks.

The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency. Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-12-06 11:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

6 Dec 2023 02:02 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable, Google Cloud Networking, Google Cloud Tasks.

The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency. Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-12-06 03:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 21:20 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Wednesday, 2023-12-06 02:30 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 17:27 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 21:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 16:20 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 17:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 14:44 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 16:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 13:52 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 15:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 12:52 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We believe the issue with Google Cloud Bigtable is partially resolved for majority of our customers.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 14:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 11:31 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

We are actively pursuing several mitigation steps currently.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-12-05 12:45 US/Pacific.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 10:57 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

Please be assured that our engineering team continues to investigate the issue for Google Cloud Bigtable and Networking components, with highest priority.

We will provide an update by Tuesday, 2023-12-05 12:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 09:53 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

Our engineering team continues to investigate the issue for Google Cloud Bigtable and Networking components.

We will provide an update by Tuesday, 2023-12-05 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 09:08 PST

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable and Google Kubernetes Engine beginning on Monday, 2023-12-04 20:00 US/Pacific.

Our investigation has identified that this was not caused due to Google Cloud Networking.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 07:51 PST

Summary: Network issue causing customers to experience elevated errors and increased latency across multiple services in us-central1

Description: We are experiencing a Google Cloud Networking issue impacting Google Cloud Bigtable and Google Kubernetes Engine beginning on Monday, 2023-12-04 20:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 09:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

5 Dec 2023 07:19 PST

Summary: Network issue causing customers to experience elevated errors and increased latency across multiple services in us-central1

Description: We are experiencing an issue with Google Cloud Networking beginning at Monday, 2023-12-04 20:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 07:50 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may experience an elevated errors and increased latency in us-central1.

Workaround: None at this time.