Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Firestore

High latency and increased error rates in northamerica multi-region.

Incident began at 2021-06-22 19:38 and ended at 2021-06-22 20:51 (all times are US/Pacific).

Date Time Description
23 Jun 2021 14:40 PDT

Incident Start: 22 June 2021 19:38

Incident End: 22 June 2021 20:51

Duration: 1 hours, 13 minutes

Affected Services and Features: Google Cloud Firestore

Regions/Zones: nam5 multi-region

Description: Google Cloud Firestore experienced elevated latency and error rates on up top 50% of projects in the nam5 multi-region for a duration of 1 hour and 13 minutes. The root cause of the issue was a traffic spike from an internal daily batch job which performs quota adjustments. This caused lock contention on the backend databases which, in conjunction with poor traffic distribution and misdirected traffic, ultimately overloaded the service.

Customer Impact:

  • Affected customers may see “Deadline Exceeded” errors and degraded performance.
22 Jun 2021 21:53 PDT

The issue with Cloud Firestore has been resolved for all affected users as of Tuesday, 2021-06-22 21:53 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
22 Jun 2021 21:26 PDT

Summary: High latency and increased error rates in northamerica multi-region.

Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific.

At the moment we see performance improvements and error rate is back to its original state. Our engineering team continues to monitor the situation.

We will provide an update by Tuesday, 2021-06-22 22:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance.

Workaround: None at this time.
22 Jun 2021 20:45 PDT

Summary: High latency and increased error rates in northamerica multi-region.

Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-06-22 21:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance.

Workaround: None at this time.