Incident Report

Summary

On Friday, 24 January 2025, AppSheet customers were unable to load AppSheet apps with the app editor or the app load page due to ‘500’ errors and timeouts. Around 60% of the requests were impacted in us-east4 and europe-west4 for a duration of 1 hour and 50 minutes.

We sincerely apologize to our Google Cloud customers for the disruption you experienced.

Root Cause

A database schema migration in production triggered a cascading incident. The migration caused failures and timeouts on the primary database, disrupting most AppSheet operations and preventing apps loading for users in us-east4 and europe-west4. The sustained outage occurred due to a surge of retries, overloading the secondary authentication database and rendering it completely unresponsive for requests to the affected regions. The authentication database is responsible for storing user authentication tokens. Traffic was migrated to the us-central1 and us-west1 regions, after which issues pertaining to user auth tokens were resolved.

However, this triggered an increase in load on our service for validating users’ Workspace license entitlements, due to that information no longer being available in cache. The request rate went up significantly, triggering aggressive load shedding, resulting in elevated latency for 95% of the traffic. This further aggravated latency after traffic migration to us-central1 and us-west1 was performed.

Remediation and Prevention

Google engineers were alerted to the outage via an automated alert on 24 January 2025 09:42 US/Pacific and immediately started an investigation. To mitigate the impact, engineers redirected the traffic from us-east4 and europe-west4, to us-central1 and us-west1. The resultant load shedding that occurred on the licensing server recovered by 11:20 US/Pacific, once we restored our authentication database and gradually reverted traffic to us-east4 and europe-west4.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Improve alerting and monitoring of on license server traffic to reduce impact on latencies when traffic migration happens.
• Gradually reduce dependency on licensing servers to avoid failures arising from either increased traffic, or unavailability of licensing servers.
• We are reviewing measures to increase the stability of our authentication database, to ensure optimal handling of any surge in requests.

Detailed Description of Impact

On Friday, 24 January 2025, from 09:30 to 11:20 US/Pacific, approximately 60% of the AppSheet requests in us-east4 and europe-west4 may have failed.

• Affected customers were unable to load AppSheet apps with the app editor or the app load page.
• Affected customers experienced elevated ‘500’ errors and timeouts.
• Some customers may also have observed intermittent latency.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 January, 2025 09:30

Incident End: 24 January, 2025 11:20

Duration: 1 hour, 50 minutes

Affected Services and Features:

AppSheet

Regions/Zones:

us-east4 and europe-west4

Description:

AppSheet experienced availability issues in us-west4 and europe-west4 for a total duration of 1 hour, 50 minutes.

From our preliminary analysis, a schema migration initiated on a backend database caused errors and request timeouts. This led to increased retries from clients, causing a secondary database that manages authentication for us-east4 and europe-west4 to become overloaded. The overload on the authentication database subsequently impacted another dependency licensing server which also became overloaded.

While the original database issues caused by the schema migration were fully resolved, the authentication and licensing servers continued to observe issues. Google engineers mitigated the overloaded authentication database by shifting traffic away from us-east4 and europe-west4 regions, resolving the issue. The licensing server recovered by 11:20 US/Pacific due to organic traffic reduction.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

• Affected customers were unable to load AppSheet apps with the editor app load page.
• Affected customers experienced elevated 500 errors and timeouts.
• Some customers may also have observed intermittent latency.

The issue with AppSheet has been resolved for all affected users as of Friday, 2025-01-24 11:20 US/Pacific.

Preliminary investigation narrowed down the trigger of the issue to be a schema migration to our backend database which caused all requests to temporarily fail with 500 errors and timeouts. This caused a large amount of traffic in us-east4 and europe-west4. The issue was fully mitigated once the migration completed.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Appsheet is unavailable in us-east4 and europe-west4

Description: Mitigation work is currently underway by our engineering team. We are showing signs of recovery and some users may observe elevated latency while we work towards full recovery.

We will provide more information by Friday, 2025-01-24 12:30 US/Pacific.

Diagnosis: Customers impacted by this issue are unable to load apps via either editor or app load page. Customers may also observe intermittent latency.

Workaround: None at this time.

Summary: Appsheet is unavailable in us-east4 and europe-west4

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2025-01-24 11:30 US/Pacific.

Diagnosis: Customers impacted by this issue are unable to load apps via either editor or app load page.

Workaround: None at this time.