Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 6 December 2023 04:20

Incident End: 6 December 2023 16:47

Duration: 12 hours, 27 minutes

Affected Services and Features:

Google App Engine

Regions/Zones: asia-east1, asia-east2, asia-northeast1, asia-northeast2, asia-northeast3, asia-south1, asia-southeast1, asia-southeast2, australia-southeast1, europe-central2, europe-west1, europe-west2, europe-west3, europe-west6, northamerica-northeast1, southamerica-east1, us-central1, us-east1, us-east4, us-west1, us-west2, us-west3, us-west4

Description: Starting on 6 December 2023 at 04:20, Google App Engine Flex experienced app unavailability issues for a duration of 12 hours and 27 minutes. This occurred due to an invalid backend service configuration update that led to L7 Load Balancer state inconsistency across regions. The issue was mitigated at 16:47 once this configuration change was rolled back.

Customer Impact:

• Google App Engine Flex deployments may have completed successfully, but the app became unavailable after some time.
• Affected customers encountered the error message "httpRequest.status=502 , resource.type="http_load_balancer", jsonPayload.statusDetails="failed_to_pick_backend"

The issue with Google App Engine has been resolved as of Wednesday, 2023-12-06 16:48 US/Pacific.

We understand that this issue has impacted your ability to access and use our services, we deeply appreciate your patience and cooperation while we worked to mitigate the issue.

We take pride in providing our customers with a reliable and highly available service,

Please rest assured of the fact that we are committed to doing everything we can to prevent issues like this from happening in the future.

Thank you for choosing us.

Summary: Google App Engine (GAE) Flex new deployments are experiencing issues with app availability.

Description: We have been experiencing an issue with Google App Engine Flex deployments beginning on Wednesday, 2023-12-06 04:20 US/Pacific.

Our engineering team rolled out mitigation to europe-west2 and a few other impacted locations globally.

The current telemetry and monitoring is showing healthy progress across all regions where the fix has been rolled out.

Our engineers continue to roll out the fix globally to ensure all locations are mitigated as quickly as possible.

We will provide an update by Wednesday, 2023-12-06 17:00 US/Pacific with current details.

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.

Diagnosis:

• For affected customers, new GAE Flex deployments may complete successfully but the app could become unavailable after some time. Affected customers will see an error message similar to “httpRequest.status=502 , resource.type="http_load_balancer", jsonPayload.statusDetails="failed_to_pick_backend".

• Applications that are running on existing deployments are unaffected.

Workaround: There is no known workaround at this time.

Summary: Google App Engine (GAE) Flex new deployments are experiencing issues with app availability.

Description: We are experiencing an issue with Google App Engine Flex deployments beginning at Wednesday, 2023-12-06 04:20 US/Pacific.

Our engineering team rolled out mitigation to europe-west2 and the current telemetry shows no further issues in that region. Our engineers are closely monitoring the region to ensure the mitigation works and are continuing to roll out mitigation in other regions.

We will provide an update by Wednesday, 2023-12-06 15:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

• For affected customers, new GAE Flex deployments may complete successfully but the app could become unavailable after some time. Affected customers will see an error message similar to “httpRequest.status=502 , resource.type="http_load_balancer", jsonPayload.statusDetails="failed_to_pick_backend".

• Applications that are running on existing deployments are unaffected.

Workaround: None at this time.

Summary: Google App Engine (GAE) Flex new deployments are experiencing issues with app availability.

Description: We are experiencing an issue with Google App Engine Flex deployments beginning at Wednesday, 2023-12-06 04:20 US/Pacific.

Our engineering team has identified a potential mitigation for the issue and is currently testing the mitigation.

We will provide an update by Wednesday, 2023-12-06 14:15 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

• For affected customers, new GAE Flex deployments may complete successfully but the app could become unavailable after some time.

• Applications that are running on existing deployments are unaffected.

Workaround: None at this time.

Summary: Google App Engine (GAE) Flex new deployments are experiencing issues with app availability.

Description: We are experiencing an issue with Google App Engine Flex deployments beginning at Wednesday, 2023-12-06 04:20 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-12-06 13:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: For affected customers, new GAE Flex deployments may complete successfully but the app could become unavailable after some time.

Workaround: None at this time.