Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Compute Engine

Global: pubsub.googleapis.com autoscaling not worked as expected

Incident began at 2021-12-07 01:56 and ended at 2021-12-13 16:59 (all times are US/Pacific).

Date Time Description
14 Dec 2021 11:59 PST

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 07 Dec 2021 01:56 PST

Incident End: 13 Dec 2021 16:59 PST

Duration: 6 Days, 15 hours, 3 minutes

Affected Services and Features:

Google Compute Engine (GCE) Autoscaler - Autoscaling using the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric.

Regions/Zones: Global

Description:

GCE autoscaling service configured to use the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric, experienced an issue while processing the autoscaling function. From preliminary analysis, the root cause was identified as code changes affecting the autoscaler policy validation that was misconfigured for the affected metric.

Customer Impact:

Customers using the metric pubsub.googleapis.com/subscription/num_undelivered_messages for autoscaling observed the autoscaling did not complete successfully. Autoscaling did not work for existing and new managed instance groups created by customers for the duration of the incident.

13 Dec 2021 17:12 PST

The issue with Google Compute Engine has been resolved for all affected users as of Monday, 2021-12-13 16:59 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

13 Dec 2021 10:52 PST

Summary: Global: pubsub.googleapis.com autoscaling not worked as expected

Description: We believe the issue with Google Compute Engine is partially resolved for previously impacted projects. Newly created projects may still experience the issue until the rollout of the fix completes.

Full resolution is expected to complete on Tuesday, 2021-12-14.

Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working.

Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

13 Dec 2021 08:21 PST

Summary: GCP engineer are investigating an issue with auto-scaling pubsub.googleapis.com may not be working as intended.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2021-12-14.

Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working.

Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

13 Dec 2021 06:48 PST

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended.

Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2021-12-13 09:15 US/Pacific with current details.

Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working.

Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

13 Dec 2021 06:26 PST

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended.

Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details.

Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working.

Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

13 Dec 2021 06:15 PST

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended.

Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details.

Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages masy see auto scaling not working.

Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.