Service Health
-
-
-
-
Incident affecting Google Compute Engine
Global: pubsub.googleapis.com autoscaling not worked as expected
Incident began at 2021-12-07 01:56 and ended at 2021-12-13 16:59 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 14 Dec 2021 | 11:59 PST | We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 07 Dec 2021 01:56 PST Incident End: 13 Dec 2021 16:59 PST Duration: 6 Days, 15 hours, 3 minutes Affected Services and Features: Google Compute Engine (GCE) Autoscaler - Autoscaling using the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric. Regions/Zones: Global Description: GCE autoscaling service configured to use the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric, experienced an issue while processing the autoscaling function. From preliminary analysis, the root cause was identified as code changes affecting the autoscaler policy validation that was misconfigured for the affected metric. Customer Impact: Customers using the metric pubsub.googleapis.com/subscription/num_undelivered_messages for autoscaling observed the autoscaling did not complete successfully. Autoscaling did not work for existing and new managed instance groups created by customers for the duration of the incident. |
| | 13 Dec 2021 | 17:12 PST | The issue with Google Compute Engine has been resolved for all affected users as of Monday, 2021-12-13 16:59 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 13 Dec 2021 | 10:52 PST | Summary: Global: pubsub.googleapis.com autoscaling not worked as expected Description: We believe the issue with Google Compute Engine is partially resolved for previously impacted projects. Newly created projects may still experience the issue until the rollout of the fix completes. Full resolution is expected to complete on Tuesday, 2021-12-14. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work. |
| | 13 Dec 2021 | 08:21 PST | Summary: GCP engineer are investigating an issue with auto-scaling pubsub.googleapis.com may not be working as intended. Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Tuesday, 2021-12-14. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work. |
| | 13 Dec 2021 | 06:48 PST | Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 09:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work. |
| | 13 Dec 2021 | 06:26 PST | Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work. |
| | 13 Dec 2021 | 06:15 PST | Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages masy see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work. |
- All times are US/Pacific