Google Cloud Service Health

Google Cloud Service Health
Incidents
High latency and increased error rates in northamerica multi-region.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Cloud Firestore

High latency and increased error rates in northamerica multi-region.

Incident began at 2021-06-22 19:38 and ended at 2021-06-22 20:51 (all times are US/Pacific).

Date	Time	Description
23 Jun 2021	14:40 PDT	Incident Start: 22 June 2021 19:38 Incident End: 22 June 2021 20:51 Duration: 1 hours, 13 minutes Affected Services and Features: Google Cloud Firestore Regions/Zones: nam5 multi-region Description: Google Cloud Firestore experienced elevated latency and error rates on up top 50% of projects in the nam5 multi-region for a duration of 1 hour and 13 minutes. The root cause of the issue was a traffic spike from an internal daily batch job which performs quota adjustments. This caused lock contention on the backend databases which, in conjunction with poor traffic distribution and misdirected traffic, ultimately overloaded the service. Customer Impact: Affected customers may see “Deadline Exceeded” errors and degraded performance.
22 Jun 2021	21:53 PDT	The issue with Cloud Firestore has been resolved for all affected users as of Tuesday, 2021-06-22 21:53 US/Pacific. We thank you for your patience while we worked on resolving the issue.
22 Jun 2021	21:26 PDT	Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. At the moment we see performance improvements and error rate is back to its original state. Our engineering team continues to monitor the situation. We will provide an update by Tuesday, 2021-06-22 22:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.
22 Jun 2021	20:45 PDT	Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-06-22 21:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.

All times are US/Pacific