Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 4 December 2023 19:50

Incident End: 6 December 2023 03:55

Duration:

• Partial mitigation: 15 hours
• Full mitigation: 1 day, 8 hours, 5 minutes

Affected Services and Features:

Google Cloud Bigtable

Regions/Zones: us-central1

Description:

Approximately 0.06% of Google Cloud Bigtable dataplane requests experienced increased latency and errors in the us-central1 region. A partial mitigation was applied 15 hours from the start of the incident and the incident was fully mitigated after 1 day, 8 hours and 5 minutes. The root cause of the issue was an isolation failure in the machines that were running Cloud Bigtable servers, this manifested primarily as a low rate of network packet loss. The change that triggered the isolation issue has been identified and rolled back.

Customer Impact:

Impacted Bigtable customers experienced increased latency and low level of connection errors in the us-central1 region.

Additional details:

The issue was first detected by automated monitoring on 4 December 2023 at 23:35 and engineers immediately started an investigation.

As part of the investigation, on 5 December 2023 at 14:40, engineers redirected some traffic away from a subset of the affected machines which partially mitigated impact.

Engineers subsequently identified the trigger, and mitigated the issue on 6 December 2023 at 03:55, which fully resolved the latency. Additionally, engineers have put in place controls to prevent recurrence.

The issue with Google Cloud Bigtable, Google Cloud Tasks has been resolved for all affected users as of Wednesday, 2023-12-06 08:23 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable, Google Cloud Tasks.

The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency. Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-12-06 11:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable, Google Cloud Networking, Google Cloud Tasks.

The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency. Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-12-06 03:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Wednesday, 2023-12-06 02:30 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 21:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 17:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However, some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

Our engineers continue to work on mitigating the impact. We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 16:30 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: The issue with Google Cloud Bigtable is resolved for the majority of our customers. However some customers that are running non-replicated clusters in us-central1 may continue to experience elevated errors and latency.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 15:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We believe the issue with Google Cloud Bigtable is partially resolved for majority of our customers.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2023-12-05 14:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

We are actively pursuing several mitigation steps currently.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-12-05 12:45 US/Pacific.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

Please be assured that our engineering team continues to investigate the issue for Google Cloud Bigtable and Networking components, with highest priority.

We will provide an update by Tuesday, 2023-12-05 12:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable beginning on Monday, 2023-12-04 20:00 US/Pacific.

The investigation from our Google engineers has confirmed that there is no impact specific to Google Kubernetes Engine.

Our engineering team continues to investigate the issue for Google Cloud Bigtable and Networking components.

We will provide an update by Tuesday, 2023-12-05 11:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Cloud Bigtable customers experiencing errors and increased latency in us-central1

Description: We are experiencing an issue with Google Cloud Bigtable and Google Kubernetes Engine beginning on Monday, 2023-12-04 20:00 US/Pacific.

Our investigation has identified that this was not caused due to Google Cloud Networking.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Network issue causing customers to experience elevated errors and increased latency across multiple services in us-central1

Description: We are experiencing a Google Cloud Networking issue impacting Google Cloud Bigtable and Google Kubernetes Engine beginning on Monday, 2023-12-04 20:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 09:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated errors and increased latency on the impacted services in us-central1.

Workaround: None at this time.

Summary: Network issue causing customers to experience elevated errors and increased latency across multiple services in us-central1

Description: We are experiencing an issue with Google Cloud Networking beginning at Monday, 2023-12-04 20:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-12-05 07:50 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may experience an elevated errors and increased latency in us-central1.

Workaround: None at this time.