Service Health

# Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 16 October, 2024 19:36

Incident End: 17 October, 2024 02:45

Duration: 7 hours, 9 minutes

Affected Services and Features:

Google Cloud and Workspace Support - Case creation, Case updates, Chat support

Regions/Zones: Global

Description:

Google Cloud and Workspace Support encountered intermittent disruptions to case creation and management for a period of 7 hours and 9 minutes.

From preliminary analysis, we believe the cause of the issue was a recent change to Google Support’s ticket persistence layer. The change granted a subsystem unbounded resource consumption, impacting user requests. Google engineering will perform a thorough root cause analysis and ensure that appropriate measures are taken to prevent a recurrence of the issue.

Customer Impact:

• Customers impacted by this incident would have noticed delays, timeouts, and other errors during case creation and while performing case updates.
• Chat support also experienced intermittent disconnects.

The issue with Google Cloud Support has been resolved for all affected users as of Thursday, 2024-10-17 04:19 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Case creation may fail intermittently with an error

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Thursday, 2024-10-17 08:00 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see issue with case creation and other case updates.

Workaround: Retrying operations may work. For Case creation in particular, customers will be redirected to a Contact Us Form (CUF) after multiple failed case creation attempts.

Summary: Case creation may fail intermittently with an error

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2024-10-17 05:00 US/Pacific.

Diagnosis: Customers impacted by this issue may see issue with case creation and other case updates.

Workaround: Retrying operations may work. For Case creation in particular, customers will be redirected to a Contact Us Form (CUF) after multiple failed case creation attempts.

Summary: Case creation may fail intermittently with an error

Description: We are experiencing an issue with Google Cloud Support.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2024-10-17 02:30 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see issue with case creation and other case updates

Workaround: Retrying operations may work. For Case creation in particular, customers will be redirected to a Contact Us Form (CUF) after multiple failed case creation attempts.

Summary: Case creation may fail intermittently with an error

Description: We are experiencing an issue with Google Cloud Support.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2024-10-17 00:30 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see issue with case creation and other case updates

Workaround: Retrying operations may work. For Case creation in particular, customers will be redirected to a Contact Us Form (CUF) after multiple failed case creation attempts.

Summary: Case creation may fail intermittently with an error

Description: We are experiencing an issue with Google Cloud Support.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-10-16 23:00 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see issue with case creation and other case updates

Workaround: Retrying operations may work. For Case creation in particular, customers will be redirected to a Contact Us Form (CUF) after multiple failed case creation attempts.