Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Monitoring, Operations

Cloud Monitoring - Uptime - 100% error rate in southamerica-east1

Incident began at 2022-09-28 15:52 and ended at 2022-09-29 09:47 (all times are US/Pacific).

Date Time Description
29 Sep 2022 14:03 PDT

MINI INCIDENT REPORT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 28 September 2022 15:52

Incident End: 29 September 2022 09:47

Duration: 17 hours, 55 minutes

Affected Services and Features:

Cloud Monitoring - Uptime Monitoring

Regions/Zones: southamerica-east1

Description:

Cloud Uptime Monitoring experienced near 100% unavailability in the affected region for a duration of 17 hours, 55 minutes. From preliminary analysis, the root cause of the issue is due to a severed link in Google’s backend network in South America. The issue was mitigated once engineers redirected traffic to alternate links. Additional review is underway to prevent potential impact in the future.

Customer Impact:

  • Public Uptime Checks were not able to egress from South America during the incident period. Affected customers would have observed partial or absent metric data from this location.
  • Private Uptime Checks were unaffected.
  • Default alert policies against Public Uptime Checks were unaffected.
  • Custom alert policies against Public Uptime Checks may have triggered if a "metric absence" condition is present in the policy (https://cloud.google.com/monitoring/alerts/concepts-indepth#partial-metric-data) and the policy is configured to only measure Uptime metrics from South America.
29 Sep 2022 06:55 PDT

The issue with Uptime has been resolved for all affected users as of Thursday, 2022-09-29 06:55 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
29 Sep 2022 06:44 PDT

Summary: Cloud Monitoring - Uptime - 100% error rate in southamerica-east1

Description: We are experiencing an issue with Uptime beginning at Wednesday, 2022-09-28 15:52 US/Pacific.

Mitigation is currently underway and we believe the issue is now partially resolved. We do not have an ETA for full resolution at this point.

We will provide more information by Thursday, 2022-09-29 08:00 US/Pacific.

Diagnosis: Uptime - a websites monitoring product - has been unavailable in southamerica-east1.

Workaround: None at this time.
29 Sep 2022 06:41 PDT

Summary: Cloud Monitoring - Uptime 100% error rate in southamerica-east1

Description: We are experiencing an issue with Uptime beginning at Wednesday, 2022-09-28 15:52 US/Pacific.

Mitigation is currently underway and we believe the issue is now partially resolved. We do not have an ETA for full resolution at this point.

We will provide more information by Thursday, 2022-09-29 08:00 US/Pacific.

Diagnosis: Uptime - a websites monitoring product - has been available in southamerica-east1.

Workaround: None at this time.