Incident Report

Summary

Cloud Shell enables all Google Cloud users to access and manage their cloud resources directly from their web browser, using a command-line interface. This service eliminates the need for users to install the Google Cloud Software Development Kits (SDK) or other tools on their local machines. Cloud Shell is implemented as a container running on a Virtual Machine (VM) in multi-tenant Google-owned projects.

To improve scalability and availability, Cloud Shell runs multiple Google Kubernetes Engine (GKE) clusters per region, and runs in multiple regions. These clusters are updated weekly.

On 4 June 2024, some users attempting to connect to Cloud Shell may have encountered connectivity issues for a duration of 3 hours 46 minutes. We apologize for the inconvenience this service disruption has caused.

Root Cause

During a routine weekly update, the Cloud Shell engineering team deployed a script that had been tested and validated in the pre-production environment. However, when executed across production GKE clusters supporting Cloud Shell, the script unintentionally enabled GKE API-side validation for custom Kubernetes resources. This validation caused GKE to reject updates to these custom resources, ultimately leading to provisioning errors in Cloud Shell. Affected customers would have received the following error message: "Cloud Shell is experiencing some issues provisioning a VM to you. Please try again in a few minutes."

Remediation and Prevention

Google engineers were alerted to the issue via an internal monitoring alert on 4 June 2024 at 10:29 US/Pacific and immediately started an investigation. Once the nature and scope of the issue were identified, engineers rapidly developed and deployed a code fix to all affected Cloud Shell GKE clusters. This fix ensured that Cloud Shell's custom resources were correctly validated, thus resolving the provisioning errors. By 14:29 on the same day, the issue was fully mitigated for all impacted users.

Google is committed to preventing a repeat of this issue by implementing stricter change control through more comprehensive policies and guidelines for manual patching to the Cloud Shell environment.

We apologize for the impact of this issue and are taking steps to address the scope and duration of this incident as well as the root cause itself. We thank you for your business.

Detailed Description of Impact

Starting on 4 June 2024 at 10:29 US/ Pacific, some customers attempting to connect to Cloud Shell may have encountered connectivity issues for a duration of 3 hours 46 minutes. Customers affected by this issue may have received the following error on Cloud Shell: “Cloud Shell is experiencing some issues provisioning a VM to you. Please try again in a few minutes.”

Mini Incident Report

We apologize for the inconvenience this service disruption/outage has caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 4 June 2024 10:29

Incident End: 4 June 2024 14:15

Duration: 3 hours 46 minutes

Affected Services and Features: Cloud Shell

Regions/Zones: Global

Description:

Starting on 4 June 2024 10:29, customers attempting to connect to Cloud Shell may have encountered connectivity issues for the duration of 3 hours 46 minutes. Based on the preliminary analysis, the root cause was identified as a recent change performed in GKE (Google Kubernetes Engine) that resulted in an API validation error. Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

Customers affected by this issue would have encountered the following error on Cloud Shell: “Cloud Shell is experiencing some issues provisioning a VM to you. Please try again in a few minutes.”

The issue with Google Cloud Shell has been resolved for all affected users as of Tuesday, 2024-06-04 14:29 US/Pacific.

We understand that this issue impacted your ability to access and use our services, we deeply appreciate your patience and cooperation while we worked to mitigate the issue.

Thank you for choosing us.

Summary: Google Cloud Shell - Connectivity Issues

Description: We are experiencing an issue with Google Cloud Shell beginning at Tuesday, 2024-06-04 11:14 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-06-04 14:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers trying to connect to Cloud Shell may not be able to do so.

Workaround: None at this time.