Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 11 August 2023 at 16:30

Incident End: 11 August 2023 at 22:27

Duration: 5 hours, 57 minutes

Affected Services and Features:

Media CDN

Regions/Zones: Global

Description:

Google Media CDN customers saw an increase in 5xx errors and failed connections, globally, for a duration of 1 hour and 30 minutes, from 16:30 until 18:00, as well as failures when attempting to make changes to their Media CDN configurations for the full duration of 5 hours and 57 minutes. From preliminary analysis, the root cause of the issue appears to be a configuration change which was mitigated by rolling back the change and rebooting the affected machines.

Customer Impact:

Media CDN customers would have seen an increase in 5xx and failed connections. Changes to any customer configurations would have failed during the time of the incident.

The issue with Media CDN has been resolved for all affected users as of Friday, 2023-08-11 22:29 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Media CDN increased rates of 5xx errors

Description: The issue with Media CDN remains partially resolved. Error rates have reduced significantly, however they have not yet returned to normal levels.

We do not have an ETA for full resolution at this point.

We will provide an update by Friday, 2023-08-11 23:30 US/Pacific with current details.

Diagnosis: Media CDN customers would have seen an increase in 5xx and failed connections that mostly resolved around 18:00 PT.

Additionally, customers may experience errors when attempting to make changes to their Media CDN configurations.

Workaround: Retrying failed Media CDN requests should succeed.

Summary: Media CDN increased rates of 5xx errors

Description: The issue with Media CDN remains partially resolved. Error rates have reduced significantly, however they have not yet returned to normal levels.

We do not have an ETA for full resolution at this point.

We will provide an update by Friday, 2023-08-11 21:30 US/Pacific with current details.

Diagnosis: Media CDN customers would have seen an increase in 5xx and failed connections that mostly resolved around 18:00 PT.

Additionally, customers may experience errors when attempting to make changes to their Media CDN configurations.

Workaround: Retrying failed Media CDN requests should succeed.

Summary: Media CDN increased rates of 5xx errors

Description: The issue with Media CDN remains partially resolved. Error rates have reduced significantly, however they have not yet returned to normal levels.

We do not have an ETA for full resolution at this point.

We will provide an update by Friday, 2023-08-11 21:30 US/Pacific with current details.

Diagnosis: Media CDN customers would have seen an increase in 5xx and failed connections that mostly resolved around 18:00 PT.

Additionally, customers may experience errors when attempting to make changes to their Media CDN configurations.

Workaround: None at this time.

Summary: Media CDN increased rates of 5xx errors

Description: We believe the issue with Media CDN is partially resolved. At this time, we are observing a significant reduction in error rates, however they have not yet returned to normal levels.

We do not have an ETA for full resolution at this point.

We will provide an update by Friday, 2023-08-11 20:30 US/Pacific with current details.

Diagnosis: Media CDN customers would have seen an increase in 5xx and failed connections that mostly resolved around 18:00 PT.

Additionally, customers may experience errors when attempting to make changes to their Media CDN configurations.

Workaround: None at this time.

Summary: Media CDN increased rates of 5xx errors

Description: Mitigation work remains underway by our engineering team. At this time, error rates appear to be dropping.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2023-08-11 20:00 US/Pacific.

Diagnosis: Customers may experience increased rates of 5xx errors or failed connections for Media CDN requests.

Workaround: None at this time.

Summary: Media CDN increased rates of 5xx errors

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2023-08-11 19:00 US/Pacific.

Diagnosis: Customers may experience increased rates of 5xx errors or failed connections for Media CDN requests.

Workaround: None at this time.

Summary: Media CDN increased rates of 5xx errors

Description: We are experiencing an issue with Media CDN beginning at Friday, 2023-08-11 13:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2023-08-11 18:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may experience increased rates of 5xx errors or failed connections for Media CDN requests.

Workaround: None at this time.