Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking

Global: Delays provisioning SSL certificates for HTTPS load balancers

Incident began at 2022-02-11 12:00 and ended at 2022-02-11 18:26 (all times are US/Pacific).

Date Time Description
14 Feb 2022 20:08 PST

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 11 February 2022 12:00

Incident End: 11 February 2022 18:26

Duration: 6 hours, 26 minutes

Affected Services and Features:

Google Cloud Networking - Global External HTTPS load balancer SSL certificates

Regions/Zones: Global

Description:

Google Cloud Networking experienced issues with global external HTTPS load balancer SSL certificate configuration updates for a duration of 6 hours, 26 minutes. This was triggered by a component rollback, which tripped a safety check that prevented changes from taking effect. The backlog required further intervention from operators to clear; when it did the issue was mitigated.

Customer Impact:

  • Customers experienced delays with initial provisioning of managed SSL certificates.
  • Customers experienced issues with programming of newly uploaded, updated, or deleted SSL certificates that were not applied to the dataplane.

Additional details:

The issue was fully resolved on 11 February 2022 at 18:26 US/Pacific after: Modifying the safety check. Supervising the pipeline until it successfully completed, removing the delta and resolving the issue.

11 Feb 2022 18:32 PST

The issue with Cloud Networking has been resolved for all affected users as of Friday, 2022-02-11 18:26 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

11 Feb 2022 17:54 PST

Summary: Global: Delays provisioning SSL certificates for HTTPS load balancers

Description: Mitigation work is still underway by our engineering team. A configuration change is rolling out as a potential fix.

We will provide more information by Friday, 2022-02-11 20:30 US/Pacific.

Diagnosis: Affected customers may experience delays when provisioning new or updated certificates.

Workaround: None at this time.

11 Feb 2022 15:18 PST

Summary: Global: Delays provisioning SSL certificates for HTTPS load balancers

Description: We are experiencing an intermittent issue with Cloud Networking beginning at Friday, 2022-02-11 12:00 US/Pacific.

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point. We apologize to all who are affected by the disruption

We will provide more information by Friday, 2022-02-11 18:00 US/Pacific.

Diagnosis: Affected customers may experience delays when provisioning new or updated certificates.

Workaround: None at this time.