Service Health

INCIDENT REPORT

Summary:

On 28 April 2022, from 07:00 to 08:32 US/Pacific, Google App Engine and Google Cloud Functions experienced increased latency and reduced availability for a duration of 1 hour and 32 minutes in one zone in the us-central1 region. Additionally, customers were unable to create, view, or search support cases in the Google Cloud Support Center and Google Admin Console for 1 hour and 24 minutes. We sincerely apologize for the impact to your service or application. We have completed an internal investigation and are taking immediate steps to improve our service’s quality and reliability.

Root Cause:

The Serverless stack relies on a file serving service for container images. The issue was triggered when the file serving component of the Serverless stack experienced a sudden increase in traffic. A bug, introduced in a recent configuration change to the file serving service, was surfaced by the sudden increase in traffic, causing many threads to get stuck. This led to resource exhaustion on the affected file servers, causing some tasks to crash and preventing new requests from completing.

Remediation and Prevention:

Google engineers were alerted to the issue on Thursday, 28 April 2022, at 07:11 US/Pacific and immediately started an investigation. Once the affected component was identified, engineers added additional capacity to the component in the zone that was experiencing the degradation. This mitigated the resource exhaustion, and the service recovered at 08:20 US/Pacific.

To prevent recurrence of the issue, engineers rolled back the configuration change to the previous stable version and implemented an automated release block to prevent any unintended release of a version that included the bug.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We are taking the following steps to prevent this or similar issues from happening again:

To improve resolution time for future issues of this type, we are calibrating our alerting system to give us an earlier and more precise notification which will allow us to diagnose the issue more quickly. We are also evaluating what kinds of load/stress tests could deterministically detect this kind of issue in the future, so that such regressions are caught automatically before they are deployed to production.

Detailed Description of Impact:

On 28th April 2022 from 7:00 PT to 8:32 PT

Google App Engine

Google App Engine experienced increased latency and reduced availability in one zone in us-central1 for a period of 1 hour and 32 minutes. Customers may have experienced increased latency or higher error rate for App Engine projects.

Google Cloud Functions

Google Cloud Functions experienced increased latency and reduced availability in one zone in us-central1 for a period of 1 hour and 32 minutes. “Google Cloud Functions” customers updating their functions (e.g. deploying a new version) may have experienced increased latency or failures, notably failing health checks.

Google Cloud Support and Google Workspace Support

Customers were unable to create, view, or search support cases in the Google Cloud Support Center or Google Admin Console for 1 hour and 24 minutes. In addition, a small number of customers had degraded access to phone support, being redirected to a queue requiring additional manual authentication with the support agent.

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 28 April 2022 07:00

Incident End: 28 April 2022 08:32

Duration: 1 hours, 32 minutes

Affected Services and Features:

Google App Engine Google Cloud Functions

Regions/Zones: us-central1

Description:

Google App Engine and Google Cloud Functions experienced increased latency and reduced availability in us-central1 for a period of 1 hour and 32 minutes. From the preliminary investigation root cause is related to a bug that caused one component of App Engine to crash under heavy load in us-central1.

Customer Impact:

Google App Engine Customers may have experienced increased latency or higher error rate for App Engine projects.

Google Cloud Functions Customers updating their functions (e.g. deploying a new version) may have experienced increased latency or failures, notably failing health checks.

The issue with Google App Engine has been resolved for all affected projects as of Thursday, 2022-04-28 08:52 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Google App Engine Increased Latency in us-central1

Description: We are experiencing an issue with Google App Engine beginning at Thursday, 2022-04-28 07:00 US/Pacific.

Our engineers believe the issue is mitigated and are validating.

We will provide an update by Thursday, 2022-04-28 09:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may have experienced increased latency for App Engine projects in us-central1.

Workaround: None at this time.

Summary: Google App Engine Increased Latency in us-central1

Description: We are experiencing an issue with Google App Engine beginning at Thursday, 2022-04-28 07:00 US/Pacific.

Our engineers believe the issue is mitigated and are validating.

We will provide an update by Thursday, 2022-04-28 09:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may have experienced increased latency for App Engine projects in uc-central1.

Workaround: None at this time.