Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 14 December 2022 09:52

Incident End: 03 January 2023 07:15

Duration: 19 days, 21 hours, 23 minutes

Affected Services and Features:

Google Cloud Console UI: Load Balancer

Regions/Zones: Global

Description:

Google Cloud Console User Interface (UI) Load Balancer customers experienced an error “Required ‘Alpha Access’ permission for ‘Compute API’” globally for 19 days, 21 hours, and 23 minutes. From preliminary analysis, the root cause of the issue appears to be due to a recent release. A fix has been rolled out, and the issue is mitigated.

Customer Impact:

Customers received an error “Required ‘Alpha Access’ permission for ‘Compute API.’” while creating/updating Load Balancer with logging enabled. Workaround was available to create Load Balancer without logging enabled and later enabling logging from gcloud CLI. Approximately 350 failed Remote Procedure Calls (RPCs) were observed to have experienced this impact for the duration of the incident.

The issue with Google Cloud Console has been resolved for all affected users as of Tuesday, 2023-01-03 08:08 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with the logging option enabled

Description: The mitigation plan has been already prepared and tested.

The mitigation is expected to complete on the beginning of January 2023.

We will provide more information by Tuesday, 2023-01-03 17:00 US/Pacific.

Diagnosis: Affected customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with logging option enabled.

Workaround: Customers can use gCloud to create or modify L7 Internal Load balancers with logging turned on. Alternatively customers can turn off logging, make any changes in the UI, then turn logging back on.

Summary: Customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with the logging option enabled

Description: We are experiencing an issue with Google Cloud Console beginning at Wednesday, 2022-12-21 05:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2022-12-22 10:00 US/Pacific with current details.

Diagnosis: Affected customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with logging option enabled.

Workaround: Customers can use gCloud to create or modify L7 Internal Load balancers with logging turned on. Alternatively customers can turn off logging, make any changes in the UI, then turn logging back on.

Summary: Customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with the logging option enabled

Description: We are experiencing an issue with Google Cloud Console beginning at Wednesday, 2022-12-21 05:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-21 10:10 US/Pacific with current details.

Diagnosis: Affected customers are unable to create or modify L7 Internal Load balancers using the Cloud Console UI with logging option enabled.

Workaround: Customers can use gCloud to create or modify L7 Internal Load balancers with logging turned on. Alternatively customers can turn off logging, make any changes in the UI, then turn logging back on.

Summary: Issues with creating and updating backend services when logging option is enabled

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 11:00 US/Pacific.

Diagnosis: Customers are unable to create/update backend services with logging option enabled.

Workaround: Customers may try to use gCloud to create/update backend services.

Summary: Issues with creating and updating backend services when logging option is enabled

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2022-12-21 11:00 US/Pacific.

We will provide more information by Wednesday, 2022-12-21 11:00 US/Pacific.

Diagnosis: Customers are unable to create/update backend services with logging option enabled.

Workaround: Customers may try to use gCloud to create/update backend services.

Summary: Issues with creating and updating backend services when logging option is enabled

Description: We are experiencing an issue with Google Cloud Console.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-21 08:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers are unable to create/update backend services with logging option enabled.

Workaround: Customers may try to use gCloud to create/update backend services.