Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking

Spectrum Access System Issues

Incident began at 2023-03-03 10:40 and ended at 2023-03-03 18:00 (all times are US/Pacific).

Date Time Description
6 Mar 2023 15:00 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Impact Period 1:

  • Incident Start: 3 March 2023 10:40
  • Incident End: 3 March 2023 12:30

Impact Period 2:

  • Incident Start: 3 March 2023 17:00
  • Incident End: 3 March 2023 18:00

Cumulative Duration: 2 hours, 50 minutes

Affected Services and Features:

Networking - Spectrum Access System

Regions/Zones: Global

Description:

Google Spectrum Access System (SAS) experienced elevated heartbeat latency and error rates for a duration of 2 hours, 50 minutes globally. From preliminary analysis, the root cause of the issue was caused by transaction failures from an upstream internal database.

Our engineering team is reviewing migrating to an alternative system to prevent a future recurrence.

Customer Impact:

Affected customer’s Citizens Broadband Radio Service Device (CBSD) experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response. If a CBSD fails to receive a successful heartbeat response before their transmitExpireTime, they lose their grant. In the event of a CBSD grant loss, additional time might have been required in order for the CBSD to successfully obtain a new grant.

Additional details:

Post-outage, if a CBSD lost their grant on 03 March 2023, it can relinquish and request a new one.
3 Mar 2023 12:57 PST

The issue with Spectrum Access System has been resolved for all affected users as of Friday, 2023-03-03 12:20 US/Pacific.

Customers should not be experiencing failing Heartbeat Requests after Friday, 2023-03-03 12:20:00 US/Pacific.

Customer CBSDs experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response.

There was no workaround during the outage. Post-outage, if a CBSD loses their grant, they can relinquish it and re-request it with the same parameters. Then the SAS should grant it to them immediately.

We thank you for your patience while we worked on resolving the issue.
3 Mar 2023 12:39 PST

Summary: Spectrum Access System Issues

Description: We've received a report of an issue with Spectrum Access System as of Friday, 2023-03-03 11:56 US/Pacific. We will provide more information by Friday, 2023-03-03 13:30 US/Pacific.

Diagnosis: None at this time.

Workaround: None at this time.