Google Cloud Service Health

Google Cloud Service Health
Incidents
us-central1: Cloud Functions experiencing increased latency and pending queue aborted error request rate.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Cloud Functions, Google App Engine

us-central1: Cloud Functions experiencing increased latency and pending queue aborted error request rate.

Incident began at 2021-09-07 11:19 and ended at 2021-09-08 11:00 (all times are US/Pacific).

Date	Time	Description
9 Sep 2021	14:09 PDT	We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 07 September 2021 11:19 Incident End: 08 September 2021 11:00 Duration: 23 hours, 41 minutes Affected Services and Features: Google App Engine - Standard Google Cloud Functions Regions/Zones: us-central1 Description: Google App Engine and Google Cloud Functions experienced elevated latency on requests in us-central1 on a small number of projects for a duration of 23 hours, 41 minutes. From preliminary analysis, the root cause of the issue is due to a sudden increase in load within the region. Customer Impact: Affected customers would have experienced elevated serving latency on apps and functions. This may have manifested with http errors and corresponding cloud logging entries such as: “Request was aborted after waiting too long to attempt to service your request.” Additional Details: Impact was mitigated by applying targeted throttling.
8 Sep 2021	12:28 PDT	The issue with Google Cloud Functions has been resolved for all affected projects as of Wednesday, 2021-09-08 12:27 US/Pacific. Please follow the link to status dashboard post with more details: https://status.cloud.google.com/incidents/uaRinwS8pu2yyjdYbDaM We thank you for your patience while we worked on resolving the issue.
8 Sep 2021	12:16 PDT	Summary: us-central1: Cloud Functions experiencing increased latency and pending queue aborted error request rate. Description: Engineering team investigation has identified increased latency and an increase in pushbacks to requests (increase in pending queue aborted error request rate) for some customers. Our engineering team continues to work on the mitigation to reduce the pushback errors and reduce the latency observed. For further updates please follow this link: https://status.cloud.google.com/incidents/uaRinwS8pu2yyjdYbDaM Diagnosis: Customer may observe increased latency and heightened pending queue aborted error request rate. Workaround: None at this time.

Date

Time

Description

9 Sep 2021

14:09 PDT

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 07 September 2021 11:19

Incident End: 08 September 2021 11:00

Duration: 23 hours, 41 minutes

Affected Services and Features:

Google App Engine - Standard
Google Cloud Functions

Regions/Zones: us-central1

Description:

Google App Engine and Google Cloud Functions experienced elevated latency on requests in us-central1 on a small number of projects for a duration of 23 hours, 41 minutes. From preliminary analysis, the root cause of the issue is due to a sudden increase in load within the region.

Customer Impact:

Affected customers would have experienced elevated serving latency on apps and functions.
This may have manifested with http errors and corresponding cloud logging entries such as: “Request was aborted after waiting too long to attempt to service your request.”

Additional Details:

Impact was mitigated by applying targeted throttling.

8 Sep 2021

12:28 PDT

The issue with Google Cloud Functions has been resolved for all affected projects as of Wednesday, 2021-09-08 12:27 US/Pacific.

Please follow the link to status dashboard post with more details: https://status.cloud.google.com/incidents/uaRinwS8pu2yyjdYbDaM

We thank you for your patience while we worked on resolving the issue.

8 Sep 2021

12:16 PDT

Summary: us-central1: Cloud Functions experiencing increased latency and pending queue aborted error request rate.

Description: Engineering team investigation has identified increased latency and an increase in pushbacks to requests (increase in pending queue aborted error request rate) for some customers.

Our engineering team continues to work on the mitigation to reduce the pushback errors and reduce the latency observed.

For further updates please follow this link: https://status.cloud.google.com/incidents/uaRinwS8pu2yyjdYbDaM

Diagnosis: Customer may observe increased latency and heightened pending queue aborted error request rate.

Workaround: None at this time.

All times are US/Pacific