Incident Report

Summary:

On Monday, 18 July 2022 at 08:30 US/Pacific, Google Cloud and Google Workspace Support customers were unable to attach files to new and existing support cases via the Cloud Console and Admin Console for a period of 5 hours and 30 minutes.

To our support customers who were impacted during this outage, we sincerely apologize, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause:

Google Cloud and Google Workspace Support customers have the ability to attach files to new or existing support cases. During the initial phase of the outage, some users experienced intermittent failures while uploading files.

During a recent effort to improve the reliability of the Cloud Support backend service, a configuration change was made to our Global Service Load Balancer (GSLB), which resulted in unavailability of the file upload service. Part of this change included migrating the servers to new addresses; however, the server that handles file attachments was not ready for the migration. The configuration change inadvertently made the file upload service used by the backend service inaccessible, resulting in file uploads failing.

Remediation and Prevention:

Once the nature and scope of the issue became clear, Google engineers quickly disabled the file attachment feature while they worked on mitigation efforts. The issue was fully mitigated on Monday, 18 July 2022 at 14:00 US/Pacific, once the GSLB configuration change was rolled back.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We are committed to the following actions to prevent this from happening again:

• Improve monitoring and alerting to detect file attachment errors.
• Improve our load balancer emergency change procedures based on known best practices within Google to reduce mitigation time.
• Improve automation testing to detect invalid configuration changes in staging environments.
• Improve internal rollout procedure for load balancer changes in order to detect rollouts that have unanticipated impact and prevent them from proceeding.

We would like to apologize for the length and severity of this incident.

Detailed Description of Impact:

Service(s) Affected:

Google Cloud Console

• 100% of users could not upload files to new or existing support cases. Support case creation was not impacted.

Google Admin Console

• 100% of users could not upload files to new or existing support cases. Support case creation was not impacted.

Mini-IR (Preliminary Incident Report)

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 18 July 2022 08:30

Incident End: 18 July 2022 14:00

Duration: 5 hours, 30 minutes

Affected Services and Features:

Google Cloud Support - Cloud Console Google Workspace Support - Admin Console

Regions/Zones: Global

Description: Google Cloud Support and Google Admin Console customers were unable to attach files to new and existing support cases for a period of 5 hours and 30 minutes. From preliminary analysis, the root cause of the issue was due to an inadvertent change to our Global Service Load Balancer (GSLB), which resulted in nonavailability of the upload service.

Customer Impact:

• Google Cloud Support: Customers were unable to attach files to new or existing support cases.
• Google Workspace Support: Customers were unable to attach files to new or existing support cases.

The issue with Google Cloud Support has been resolved for all affected users as of Monday, 2022-07-18 14:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Cloud Support Systems File upload failure

Description: The case attachment upload feature has been disabled for all Cloud Console users.

Engineering team is currently working on a mitigation and will enable the feature once fully resolved.

We do not have a mitigation ETA for mitigation at this point.

We will provide more information by Monday, 2022-07-18 14:30 US/Pacific.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature has been disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: The case attachment upload feature has been disabled for all Cloud Console users.

Engineering team is currently working on a mitigation and will enable the feature once fully resolved.

We do not have a mitigation ETA for mitigation at this point.

We will provide more information by Monday, 2022-07-18 13:30 US/Pacific.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature has been disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: The case attachment upload feature has been disabled for all Cloud Console users.

Engineering team is investigating the root cause of the file upload feature and will enable it back once resolved.

We will provide an update by Monday, 2022-07-18 12:35 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature has been disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: The case attachment upload feature has been disabled for all Cloud Console users.

Engineering team is investigating the root cause of the file upload feature and will enable it back once resolved.

We will provide an update by Monday, 2022-07-18 11:35 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature has been disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: The case attachment upload feature has been disabled for all Cloud Console users.

Engineering team is investigating the root cause of the file upload feature and will enable it back once resolved.

We will provide an update by Monday, 2022-07-18 11:30 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature has been disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: Engineering team is currently investigating the root cause of the issue.

The case attachments feature will be disabled for all Cloud Console users in the next 30 minutes. This mitigation step is being taken to enable the customer to utilize the Cloud support feature while the issue is being investigated.

We will provide an update by Monday, 2022-07-18 10:30 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads. The case attachment feature will be disabled until the issue is resolved.

Workaround: None at this time.

Summary: Google Cloud Support Systems File upload failure

Description: We are experiencing an issue with Google Cloud Support.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2022-07-18 10:12 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads

Workaround: None

Summary: Goggle Cloud Support Systems File upload failure

Description: We are experiencing an issue with Google Cloud Support.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2022-07-18 10:12 US/Pacific with current details.

Diagnosis: Users experiencing failure with file uploads

Workaround: None