Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 22 April 2024 09:00

Incident End: 22 April 2024 22:18

Duration: 13 hours, 18 minutes

Affected Services and Features:

Google Batch (new batch job creation, scheduling of queued jobs)

Regions/Zones: us-central1

Description:

Google Batch experienced an issue with almost all new incoming jobs stuck in Queued state in the us-central1 region for a period of 8 hours, 37 minutes.

From our preliminary analysis, the root cause of the issue is increased transaction contention resulting in significant latency increase. This effect accumulated and further slowed down the system's processing and resulted in the jobs to remain in Queued status instead of progressing to Scheduled status.

Google engineers were alerted by our internal monitoring and immediately started an investigation. Once the nature of impact was clear, our engineering team mitigated the impact by disabling new job scheduling, and cleared the stuck jobs by restarting them.

Customer Impact: Customers from the affected region trying to use Batch experienced delays while changing the status of their batch jobs from Queued to Scheduled. As a workaround, impacted customers were advised to try submitting their jobs from another region wherever possible.

The issue with Batch has been resolved for all affected users as of Monday, 2024-04-22 22:18 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Batch - Service Issues in us-central1

Description: We've received a report of an issue with Batch as of Monday, 2024-04-22 09:00 US/Pacific.

This issue is impacting all new incoming requests for batch jobs in the 'us-central1' region.

Our engineering team continues to work on the mitigation strategy identified. There is no ETA for the completion of the mitigation activities.

We will provide more information by Monday, 2024-04-22 23:30 US/Pacific.

Diagnosis: Customers trying to use Batch would experience delays while changing the status of their batch jobs from Queued to Scheduled.

Workaround: Customers could try submitting their jobs from another region as a workaround.

Summary: Batch - Service Issues in us-central1

Description: We've received a report of an issue with Batch as of Monday, 2024-04-22 09:00 US/Pacific.

This issue is impacting all new incoming requests for batch jobs in us-central1.

Our engineering team continues to work on the mitigation strategy identified. There is no ETA for the completion of the mitigation activities.

We will provide more information by Monday, 2024-04-22 21:30 US/Pacific.

Diagnosis: Customers trying to use Batch would experience delays while changing the status of their batch jobs from Queued to Scheduled.

Workaround: Customers could try submitting their jobs from another region as a workaround.

Summary: Batch - Service Issues in us-central1

Description: We've received a report of an issue with Batch as of Monday, 2024-04-22 09:00 US/Pacific.

This issue is impacting all new incoming requests for batch jobs in us-central1.

Our engineering team has identified a mitigation strategy. The mitigation work is currently underway.

We will provide more information by Monday, 2024-04-22 18:30 US/Pacific.

Diagnosis: Customers trying to use Batch would experience delays while changing the status of their batch jobs from Queued to Scheduled.

Workaround: Customers could try submitting their jobs from another region as a workaround.

Summary: Batch - Service Issues in us-central1

Description: We've received a report of an issue with Batch as of Monday, 2024-04-22 09:00 US/Pacific.

This issue is impacting all new incoming requests for batch jobs in us-central1.

Our engineering team continues to investigate the issue in hand to ascertain a mitigation plan.

We will provide more information by Monday, 2024-04-22 16:30 US/Pacific.

Diagnosis: Customers trying to use Batch would experience delays while changing the status of their batch jobs from Queued to Scheduled.

Workaround: Customers could try submitting their jobs from another region as a workaround.

Summary: Batch - Service Issues in us-central1

Description: We've received a report of an issue with Batch as of Monday, 2024-04-22 09:00 US/Pacific.

This issue is impacting all new incoming requests for batch jobs in us-central1.

Our engineering team is investigating the issue in hand to ascertain a mitigation plan.

We will provide more information by Monday, 2024-04-22 15:15 US/Pacific

Diagnosis: Customers trying to use Batch would experience delays while changing the status of their batch jobs from Queued to Scheduled.

Workaround: Customers could try submitting their jobs from another region as a workaround.