Google Cloud Service Health

Google Cloud Service Health
Incidents
We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Operations, Cloud Monitoring, Cloud External Key Manager

We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific.

Incident began at 2021-06-21 12:21 and ended at 2021-06-21 21:55 (all times are US/Pacific).

Date	Time	Description
22 Jun 2021	15:02 PDT	Incident Start: 21 June 2021 12:01 Incident End: 21 June 2021 21:55 Duration: 9 hours, 54 minutes Affected Services and Features: Google Cloud Monitoring, Cloud EKM Regions/Zones: Global Description: Google Cloud Monitoring experienced intermittent API errors with a gradually increasing error rate on a limited set of internal endpoints for 9 hours and 54 minutes. The Cloud Monitoring section in the Cloud Console experienced intermittent errors when loading pages due to the underlying API errors. This included the following pages; Monitoring Homepage, Dashboard Builder, GKE Dashboard, Metrics Explorer, Network Topology, Uptime Checks Service. Additionally, attempting to create the first Cloud EKM key [1] in a project would fail during this period. If a project already has (or at some point had) Cloud EKM keys, they can continue to create keys in those projects and use them. The root cause of the issue is suspected to be the rollout of a permission change which incorrectly restricted access to the underlying database which stores metadata information about the affected APIs. Customer Impact: Cloud Console errors on up to 10% of the listed page loads or equivalent API’s. Additional details: Workaround: Retrying failed requests with exponential backoff returned successful results for some customers. Workaround: Before attempting to create the first Cloud EKM key, running gcloud beta services identity create --service=cloudkms.googleapis.com --project $KEY_PROJECT_ID” prevented the failure. Reference(s): [1] https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services/generateServiceIdentity
21 Jun 2021	21:46 PDT	The issue with Cloud Monitoring has been resolved for all affected projects as of Monday, 2021-06-21 21:46 US/Pacific. We thank you for your patience while we worked on resolving the issue.
21 Jun 2021	21:29 PDT	Summary: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Description: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-06-21 23:00 US/Pacific with current details. Diagnosis: You should be able to see an API error about "method not available" in the Network Tab in your browser's developer console. Workaround: Trying to refresh or re-issue the query might help.
21 Jun 2021	20:26 PDT	Summary: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Description: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-06-21 21:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: You should be able to see an API error about "method not available" in the Network Tab in your browser's developer console. Workaround: Trying to refresh or re-issue the query might help.

All times are US/Pacific