Incident affecting Google Cloud Networking
Spectrum Access System Issues
Incident began at 2023-03-03 10:40 and ended at 2023-03-03 18:00 (all times are US/Pacific).
|6 Mar 2023
Mini Incident Report
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.
(All Times US/Pacific)
Impact Period 1:
Impact Period 2:
Cumulative Duration: 2 hours, 50 minutes
Affected Services and Features:
Networking - Spectrum Access System
Google Spectrum Access System (SAS) experienced elevated heartbeat latency and error rates for a duration of 2 hours, 50 minutes globally. From preliminary analysis, the root cause of the issue was caused by transaction failures from an upstream internal database.
Our engineering team is reviewing migrating to an alternative system to prevent a future recurrence.
Affected customer’s Citizens Broadband Radio Service Device (CBSD) experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response. If a CBSD fails to receive a successful heartbeat response before their transmitExpireTime, they lose their grant. In the event of a CBSD grant loss, additional time might have been required in order for the CBSD to successfully obtain a new grant.
Post-outage, if a CBSD lost their grant on 03 March 2023, it can relinquish and request a new one.
|3 Mar 2023
The issue with Spectrum Access System has been resolved for all affected users as of Friday, 2023-03-03 12:20 US/Pacific.
Customers should not be experiencing failing Heartbeat Requests after Friday, 2023-03-03 12:20:00 US/Pacific.
Customer CBSDs experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response.
There was no workaround during the outage. Post-outage, if a CBSD loses their grant, they can relinquish it and re-request it with the same parameters. Then the SAS should grant it to them immediately.
We thank you for your patience while we worked on resolving the issue.
|3 Mar 2023
Summary: Spectrum Access System Issues
Description: We've received a report of an issue with Spectrum Access System as of Friday, 2023-03-03 11:56 US/Pacific. We will provide more information by Friday, 2023-03-03 13:30 US/Pacific.
Diagnosis: None at this time.
Workaround: None at this time.
- All times are US/Pacific