Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google BigQuery

BigQuery S3 Data Transfer Service experiencing issues in us-multiregion, asia-noutheast1 and asia-southeast1

Incident began at 2022-02-21 21:45 and ended at 2022-02-25 12:22 (all times are US/Pacific).

Date Time Description
25 Feb 2022 15:20 PST

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 21 February 2022 21:45

Incident End: 25 February 2022 12:22

Duration: 3 days, 14 hours, 37 minutes

Affected Services and Features:

Google BigQuery - Data Transfer Service

Regions/Zones: Global

Description:

Google BigQuery S3 Data Transfer Service experienced table validation failures globally, with the majority of impact observed in us-multiregion, asia-northeast1, and asia-southeast1. From preliminary analysis, the root cause of the issue was a rollout that started on 21 February 2022 at 01:00 US/Pacific. A rollback was initiated on 25 February 2022 at 08:29 to mitigate the issue in the most heavily affected regions (us-multiregion, asia-northeast1, and asia-southeast1) which was completed on 25 February 2022 12:01.

Customer Impact:

  • Transfer Runs for Google BigQuery S3 Data Transfer Service [1] will fail with an INVALID_ARGUMENT error if the following conditions are met:
    • The destination table in the Transfer Configuration contains a runtime parameter [2].
    • The rollback has not completed in the given region. Regions where we identified impact have been prioritized and rolled back already.

[1] - https://cloud.google.com/bigquery-transfer/docs/s3-transfer-intro

[2] - https://cloud.google.com/bigquery-transfer/docs/s3-transfer-parameters

Additional Details:

  • The rollback should complete across all regions by 01 March 2022.
  • Once the rollback completes, affected customers can retry the failed Transfer Runs to load any missed files.
25 Feb 2022 12:04 PST

The issue with Google BigQuery has been resolved for all affected users as of Friday, 2022-02-25 12:03 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
25 Feb 2022 09:57 PST

Summary: BigQuery S3 Data Transfer Service experiencing issues in us-multiregion, asia-noutheast1 and asia-southeast1

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by the following dates/times: US-Multiregion - Friday, 2022-02-25 12:00 US/Pacific. Asia-southeast1 and Asia-northeast1 - Friday, 2022-02-25 14:00 US/Pacific. Other regions potentially impacted - Tuesday, 2022-03-01 EOB US/Pacific.

We will provide more information by Friday, 2022-02-25 12:00 US/Pacific.

Diagnosis: S3 transfers may fail table validation when table name contains template argument (e.g., "{run_time}")

Workaround: None at this time.