Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud Platform. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit cloud.google.com.

Google Cloud Support Incident #19002

Issue with Google Cloud Support

Incident began at 2019-05-17 10:08 and ended at 2019-05-17 14:07 (all times are US/Pacific).

Date Time Description
May 31, 2019 14:55

ISSUE SUMMARY

On Friday 17 May, 2019, Google Cloud Support experienced case creation and read issues for a duration of 4 hours. We sincerely apologize to our customers for the difficulties and delays in communicating with Google Cloud Support during this time.

DETAILED DESCRIPTION OF IMPACT

On Friday 17 May, 2019 from 10:08 to 14:07 US/Pacific, both Google Cloud Support and our customers were unable to view, create, and modify Support cases, utilize Chat Support, or view known issues via the Google Cloud Support Center (GCSC) and the Cloud Console. Phone support was also unavailable from 10:08 to 14:26.

ROOT CAUSE

A critical external dependency for Google Cloud Support encountered an issue resulting in service unavailability, which impacted both Google’s and Google Cloud Platform customers’ ability to use the service. This impacted case creation for chat, phone, and email cases, which all rely on the external dependency to reference customer data and route to the correct channel. Unfortunately, our tooling also did not allow for an automated mitigation.

REMEDIATION AND PREVENTION

Google’s engineers were alerted to the problem at 10:26 and escalated immediately. Once the nature and scope of the situation became clear, Cloud Support identified customer communication options through secondary channels. Customers were able to file cases through the backup “Contact Support” mechanism that was automatically available in the Google Cloud Support Center (GCSC). All cases reported through the backup mechanism were routed internally to the appropriate Support Engineers.

We will refactor Cloud Support’s Business Continuity Plan to account for issues of this nature, to ensure that: - Existing cases can continue to be viewed and updated - Chat Support will be available - Phone support will be available

ISSUE SUMMARY

On Friday 17 May, 2019, Google Cloud Support experienced case creation and read issues for a duration of 4 hours. We sincerely apologize to our customers for the difficulties and delays in communicating with Google Cloud Support during this time.

DETAILED DESCRIPTION OF IMPACT

On Friday 17 May, 2019 from 10:08 to 14:07 US/Pacific, both Google Cloud Support and our customers were unable to view, create, and modify Support cases, utilize Chat Support, or view known issues via the Google Cloud Support Center (GCSC) and the Cloud Console. Phone support was also unavailable from 10:08 to 14:26.

ROOT CAUSE

A critical external dependency for Google Cloud Support encountered an issue resulting in service unavailability, which impacted both Google’s and Google Cloud Platform customers’ ability to use the service. This impacted case creation for chat, phone, and email cases, which all rely on the external dependency to reference customer data and route to the correct channel. Unfortunately, our tooling also did not allow for an automated mitigation.

REMEDIATION AND PREVENTION

Google’s engineers were alerted to the problem at 10:26 and escalated immediately. Once the nature and scope of the situation became clear, Cloud Support identified customer communication options through secondary channels. Customers were able to file cases through the backup “Contact Support” mechanism that was automatically available in the Google Cloud Support Center (GCSC). All cases reported through the backup mechanism were routed internally to the appropriate Support Engineers.

We will refactor Cloud Support’s Business Continuity Plan to account for issues of this nature, to ensure that: - Existing cases can continue to be viewed and updated - Chat Support will be available - Phone support will be available
May 17, 2019 14:16

The issue with users being unable to create new cases has been resolved for all users. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence

The issue with users being unable to create new cases has been resolved for all users. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence
May 17, 2019 10:37

We are experiencing an issue with Google Cloud Support in which users are unable to create new support cases.

We are experiencing an issue with Google Cloud Support in which users are unable to create new support cases.