Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Pub/Sub

Multiple regions completely blocked for subscribe for Pubsub

Incident began at 2025-01-08 06:54 and ended at 2025-01-08 08:07 (all times are US/Pacific).

Previously affected location(s)

Mumbai (asia-south1)Delhi (asia-south2)Jakarta (asia-southeast2)Belgium (europe-west1)Berlin (europe-west10)Doha (me-central1)Iowa (us-central1)South Carolina (us-east1)Columbus (us-east5)Dallas (us-south1)

Date Time Description
8 Jan 2025 13:23 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 8 January 2025 6:54

Incident End: 8 January 2025 8:07

Duration: 1 hour, 13 minutes

Affected Services and Features:

  • Google Cloud Pub/Sub
  • Cloud Logging
  • BigQuery Data Transfer Service

Regions/Zones: europe-west10, asia-south1, europe-west1, us-central1, asia-southeast2, us-east1, us-east5, asia-south2, us-south1, me-central1

Customers publishing from other regions may have also experienced the issue if the message storage policies [1] are set to store and process the messages in the above-mentioned regions.

Description:

Google Cloud Pub/Sub experienced a service outage in multiple regions for a duration of 1 hour and 13 minutes resulting in customers unable to publish or subscribe to the messages.

From preliminary analysis, the root cause of the issue was a configuration change which was rolled back to restore the service. Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

  • Google Cloud Pub/Sub : Customers were unable to publish or subscribe to the messages in the impacted regions. Publishing the messages from other regions may also have been impacted, if they have any of the impacted regions in their message storage policies. Backlog stats metric might be stale or missing.

  • Google BigQuery Data Transfer Service : Customers experienced failures with data transfers runs.

  • Cloud Logging : All Cloud Logs customers exporting logs to Cloud Pub/Sub experienced a delay in the log export for a duration of 26 minutes.

Reference(s):

[1] https://cloud.google.com/pubsub/docs/resource-location-restriction#message_storage_policy_overview

8 Jan 2025 08:12 PST

The issue with Google Cloud Pub/Sub has been resolved for all affected projects as of Wednesday, 2025-01-08 08:07 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

8 Jan 2025 08:03 PST

Summary: Multiple regions completely blocked for subscribe for Pubsub

Description: We are experiencing an issue with Google Cloud Pub/Sub, across multiple regions affecting publish and subscribe.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2025-01-08 09:15 US/Pacific with current details.

Diagnosis: Customers in the impacted regions are unable to subscribe to messages

Workaround: None at this time.

8 Jan 2025 07:56 PST

Summary: Multiple regions completely blocked for publish for Pubsub

Description: We are experiencing an issue with Google Cloud Pub/Sub, across multiple regions affecting publish and subscribe.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2025-01-08 09:00 US/Pacific with current details.

Diagnosis: Customers in the impacted regions are unable to publish messages

Workaround: None at this time.