Google Cloud Service Health Updates2024-03-28T17:35:02+00:00Google Cloudhttps://status.cloud.google.com/UPDATE: Cloud Customers may experience minimal service disruption in us-east5-ctag:status.cloud.google.com,2024:feed:zzizj3QAAaGqyqvwKcrT.oUH2MyMMmksqhd3sNWsj2024-03-28T17:35:02+00:00<p> Incident began at <strong>2024-03-28 09:48</strong> <span>(all times are <strong>US/Pacific</strong>).</span></p><div class="cBIRi14aVDP__status-update-text"><p>Summary: Cloud Customers may experience minimal service disruption in us-east5-c</p>
<p>Description: We are experiencing an issue with Google Compute Engine, Persistent Disk, and Google Kubernetes Engine. Our engineers are working on performing maintenance in the us-east5-c zone to minimize impact.</p>
<p>We will provide an update by Thursday, 2024-03-28 11:05 US/Pacific with current details.</p>
<p>Diagnosis: - Customers may experience increased latency or packet loss.</p>
<ul>
<li>Newly created VMs will be served via other zones in the us-east5 region.</li>
<li>The current impact is limited to existing VMs where they may experience elevated network latencies and packet loss.</li>
<li>Existing workload should not be impacted. The impact should be limited to VM-to-VM traffic.</li>
</ul>
<p>Workaround: None at this time.</p>
</div><hr><p>Affected products: Virtual Private Cloud (VPC), Google Compute Engine, Google Kubernetes Engine, Persistent Disk, Google Cloud Networking</p><p>Affected locations: Columbus (us-east5)</p>RESOLVED: Cloud Workstations service issues for us-central1 and us-west1regions.tag:status.cloud.google.com,2024:feed:TZMCWaQRtAmX9ZDVwYUx.BaMbTKiKD2QvLBzAgfsA2024-03-27T18:16:18+00:00<p> Incident began at <strong>2024-03-26 12:50</strong> and ended at <strong>2024-03-26 17:30</strong> <span>(all times are <strong>US/Pacific</strong>).</span></p><div class="cBIRi14aVDP__status-update-text"><h1>Mini Incident Report</h1>
<p>We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below.</p>
<p>Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using <a href="https://cloud.google.com/support">https://cloud.google.com/support</a> or to Google Workspace Support using the help article <a href="https://support.google.com/a/answer/1047213">https://support.google.com/a/answer/1047213</a>.</p>
<p>(All Times US/Pacific)</p>
<p><strong>Incident Start:</strong> 26 March 2024 12:50</p>
<p><strong>Incident End:</strong> 26 March, 2024 17:30</p>
<p><strong>Duration:</strong> 4 hours, 40 minutes</p>
<p><strong>Affected Services and Features:</strong></p>
<p>Cloud Workstations</p>
<p><strong>Regions/Zones:</strong></p>
<p>us-west1, us-central1</p>
<p><strong>Description:</strong></p>
<p>Google Cloud Workstations in us-west1 and us-central1 experienced 500 internal errors and slow load time in the Cloud Console for a duration of 4 hours 40 minutes. Our preliminary analysis shows the root cause of the issue being a transient overload of our API servers.</p>
<p>The traffic overload was due to a configuration change which was quickly identified and rolled back.</p>
<p>Google engineers mitigated the issue by throttling traffic as required to normalize resource utilization and serve all requests.</p>
<p><strong>Customer Impact:</strong></p>
<ul>
<li>Impacted customers would have observed 500 internal errors.</li>
<li>Experienced slow load times in the Cloud Console.</li>
</ul>
</div><hr><p>Affected products: Cloud Workstations</p><p>Affected locations: Iowa (us-central1), Oregon (us-west1)</p>