Incident Report

Summary

Beginning at 10:00 US/Pacific on Monday, 13 November 2023, Cloud IAM began a cutover from one configuration management system to another. Given Google Cloud progressively and incrementally rolls out changes, a growing percentage of users experienced authentication issues in flows using JSON Web Tokens (JWTs). In some circumstances, credentials signed with newer keys could not be verified during the rollout. As time progressed, new credentials were created and the impact increased, especially for flows that crossed regional boundaries.

On Wednesday, November 15th, starting at 12:00 PM PST, users of Firebase Authentication, Google Cloud Identity Platform (GCIP), and Firebase Security Rules in the Firestore nam5 multi-region encountered incorrect permission denied errors when trying to access services like Cloud Firestore and Cloud Storage for Firebase. This issue persisted for 6 hours and 41 minutes and was resolved at 18:41 PM PST on the same day.

To our Firebase customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The identity system uses public-key cryptography for signing and verifying identity credentials (called OpenID Connect JWTs). For security reasons, the keys are rotated frequently, and new certificates are distributed to clients in every region over a period of time. The key rotation strategy is optimized both for security and reliability. There is a time interval where both keys are available for verification while the old keys are being aged out of the signing flows.

Due to a bug in the key distribution system, the signing flows switched to using new keys before the corresponding new verification keys were distributed in all regions. Depending on client traffic patterns, some credentials signed with new keys were routed to these out-of-sync regions, causing them to be rejected. While this created an availability issue, the systems behaved appropriately by rejecting the out-of-sync keys, and at no point was security compromised.

Remediation and Prevention

Google engineers were alerted from support cases on 15 November and restored the services at 18:41 US/Pacific

The issue was mitigated by completing the rollout of new verification keys to us-central1 to reach a consistent configuration.

We apologize for the length and severity of this incident. We are taking the appropriate steps to prevent a recurrence and improve reliability in the future. Google is also completing the following actions:

• Review, identify, and resolve underlying configuration rollouts process issues to make sure configuration changes are consistent before being made visible to customers.
• Enhance monitoring systems to detect any customer-impacting changes prior to customers experiencing elevated error rates (using customer perspective).
• We are reducing the rate of changes to our Cloud authentication systems to reduce the probability of additional issues while we work to enhance our monitoring systems.

Detailed Description of Impact

Cloud IAM:

Other Google Cloud services using Cloud IAM service accounts may have received unexpected invalid credentials or 403 responses. Specifically, services that generate signed web tokens (for a service account) using one instance/task for Cloud IAM and validate that token using an instance/task using a different version during a rollout could be considered invalid unintentionally.

Firestore:

Approximately 20% of Firestore access checks were failing with invalid token errors globally. Users in the multiregion nam5 saw up to 90% of invalid token errors for Firestore Streaming APIs.

Firebase:

Firebase Auth/Identity Platform returned a 0.6% error rate due to this incident on the GetAccountInfo API and a 50% error rate on the low traffic CreateSessionCookie API. There was also a 2.6x increase in SignInWithCustomToken client error rate. Users might have experienced these as authentication failures (depending on application logic).

Cloud Storage for Firebase would have also experienced incorrect 403s when fetching files from buckets in us-central1.

Mini Incident Report

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 15 November 2023 12:00

Incident End: 15 November 2023 18:42

Duration: 6 hours, 42 minutes

Affected Services and Features:

• Firebase Authentication
• Google Cloud Identity Platform
• Firebase Security Rules
• Cloud Storage for Firebase
• Cloud Firestore

Regions/Zones: nam5 multi-region

Description:

Customers of Firebase Authentication, Google Cloud Identity Platform (GCIP) and Firebase Security Rules in the nam5 multi-region experienced incorrect permission denied errors for a duration of 6 hours and 42 minutes while attempting to authenticate access to services including Cloud Firestore and Cloud Storage for Firebase.

From preliminary analysis, the root cause of the issue was a mismatch between configurations. The mismatch impacted authentication services provided by Firebase Authentication and Google Cloud Identity Platform (GCIP), which allow access to downstream Firebase services. A dependency shared by both these services changed the mechanism for how security keys are rotated. A mismatch in timestamps between configurations led these services to invalidate some tokens. The issue was mitigated by accelerating the ongoing rollout to remaining clusters which resolved the timestamp mismatch.

Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

• Customers with Cloud Firestore databases residing in nam5 multi-region and using Firebase Security Rules to authenticate access for Cloud Firestore experienced permission denied errors when attempting to perform database operations via the Realtime Updates APIs.
• Cloud Firestore customers using IAM to authenticate access were not affected by the issue.

The issue with Cloud Firestore, Identity and Access Management, Identity Platform has been resolved for all affected users as of Wednesday, 2023-11-15 18:42 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2023-11-15 19:30 US/Pacific.

We will provide more information by Wednesday, 2023-11-15 19:40 US/Pacific.

Diagnosis: Customers may experience permission denied errors when using Firebase Rules to authenticate access to Cloud Firestore.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Our engineering team has identified a potential root cause and is working towards identifying a mitigation strategy.

Upon further investigation, our engineering team identified that the issue is limited to Cloud Firestore databases residing in nam5 multi-region.

We will provide an update by Wednesday, 2023-11-15 19:00 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors when using Firebase Rules to authenticate access to Cloud Firestore.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Our engineering team has identified a potential root cause and is working towards identifying a mitigation strategy.

Upon further investigation, our engineering team identified that the issue is limited to Cloud Firestore databases residing in nam5 multi-region.

We will provide an update by Wednesday, 2023-11-15 17:50 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors when using Firebase Rules to authenticate access to Cloud Firestore.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Our engineering team has identified a potential root cause and is working towards identifying a mitigation strategy.

Upon further investigation, our engineering team identified that the issue is limited to Cloud Firestore databases residing in nam5 multi-region.

We will provide an update by Wednesday, 2023-11-15 17:20 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors when using Firebase Rules to authenticate access to Cloud Firestore.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Initial mitigation attempt did not resolve the issue and our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Wednesday, 2023-11-15 16:50 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors when using Firebase Rules to authenticate access to Cloud Firestore.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-11-15 16:30 US/Pacific.

Diagnosis: Customers using mobile or browser clients to read or write data may experience permission denied errors.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: We are experiencing an issue with Cloud Firestore beginning at Wednesday, 2023-11-15 12:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-11-15 16:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers using mobile or browser clients to read or write data may experience permission denied errors.

Workaround: None at this time.

Summary: Some Cloud Firestore customers experiencing permission denied errors when reading and writing to their database.

Description: We are experiencing an issue with Cloud Firestore beginning at Wednesday, 2023-11-15 12:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-11-15 16:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers using mobile or browser clients to read or write data may experience permission denied errors.

Workaround: None at this time.

Summary: We are experiencing an issue with Cloud Firestore

Description: We are experiencing an issue with Cloud Firestore.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-11-15 15:30 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors, missing or insufficient permissions.

Workaround: None at this time.

Summary: We are experiencing an issue with Cloud Firestore

Description: We are experiencing an issue with Cloud Firestore.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-11-15 15:00 US/Pacific with current details.

Diagnosis: Customers may experience permission denied errors, missing or insufficient permissions.

Workaround: None at this time.