Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 3 June, 2024 20:43

Incident End: 4 June, 2024 01:27

Duration: 4 hours, 44 minutes

Affected Services and Features:

Google Cloud Task Cloud Scheduler

Regions/Zones: us-central1

Description:

Google Cloud Tasks and Cloud Scheduler experienced an increased error rate in the us-central1 region for a duration of 4 hours, 44 minutes. The issues affected Tasks API usage, Task execution, and scheduled job execution, resulting in task processing delays or failures. This may have caused delays or failures in other dependent processes. While the exact cause is still being investigated, initial findings point to a change activity causing an unexpected impact.

Customer Impact:

Increased error rate when using Tasks API and for Task execution Increased failure rate for some Scheduler jobs execution

Additional details:

There is no immediate risk for the issue to reoccur. Additional preventative action items will be identified once a complete root cause analysis is completed.

The issue with Google Cloud Scheduler, Google Cloud Tasks has been resolved for all affected users as of Tuesday, 2024-06-04 01:28 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Cloud Task and Cloud Scheduler are experiencing elevated errors while executing the task APIs in the us-central1 region.

Description: Our engineering teams are actively working to implement the mitigation fix.

The mitigation has been completed for the Google Cloud Scheduler, Customer would see the service recovery.

Our initial mitigation efforts have successfully restored partial functionality of Google Cloud Task for most customers. We anticipate full recovery by Tuesday, 2024-06-04 01:30 US/Pacific as we complete the remaining mitigation rollouts.

We will provide more information by Tuesday, 2024-06-04 02:00 US/Pacific.

Diagnosis: The impacted customers in the us-central1 region would experience increased errors when using Tasks API and for Task execution. This may result in tasks not being processed as expected, leading to delays or failures in dependent processes. Additionally, there were failures for some scheduler jobs execution which is not mitigated.

Workaround: None at this time.

Summary: Google Cloud Task and Cloud Scheduler are experiencing elevated errors while executing the task APIs in the us-central1 region.

Description: Our engineering teams are actively working to implement the mitigation fix.

The mitigation has been completed for the Google Cloud Scheduler, Customer would see the service recovery.

The mitigation for Google Cloud Task is underway and expected to be completed by Tuesday, 2024-06-04 00:15 US/Pacific. We will provide more information by Tuesday, 2024-06-04 00:45 US/Pacific.

Diagnosis: The impacted customers in the us-central1 region would experience increased errors when using Tasks API and for Task execution. This may result in tasks not being processed as expected, leading to delays or failures in dependent processes. Additionally, there were failures for some scheduler jobs execution which is not mitigated.

Workaround: None at this time.

Summary: Google Cloud Task is experiencing elevated errors while executing the task APIs in the us-central1 region.

Description: Our engineering teams are actively working to implement the mitigation fix.

The mitigation fix is expected to be completed by Tuesday, 2024-06-04 00:00 US/Pacific.

We will provide more information by Tuesday, 2024-06-04 00:30 US/Pacific.

Diagnosis: The impacted customers in the us-central1 region would experience increased errors when using Tasks API and for Task execution. This may result in tasks not being processed as expected, leading to delays or failures in dependent processes.

Workaround: None at this time.