Google Cloud Service Health

Google Cloud Service Health
Incidents
User timers broken for certain Dataflow streaming pipelines in multiple locations

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Cloud Dataflow

User timers broken for certain Dataflow streaming pipelines in multiple locations

Incident began at 2022-08-18 07:19 and ended at 2022-08-19 10:03 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Salt Lake City (us-west3)Las Vegas (us-west4)

Date	Time	Description
19 Aug 2022	10:03 PDT	The issue with Google Cloud Dataflow has been resolved for all affected users as of Friday, 2022-08-19 09:41 US/Pacific. We thank you for your patience while we worked on resolving the issue.
19 Aug 2022	08:08 PDT	Summary: User timers broken for certain Dataflow streaming pipelines in multiple locations Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2022-08-19 11:00 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	21:33 PDT	Summary: User timers broken for certain Dataflow streaming pipelines in multiple locations Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2022-08-19 08:30 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	20:19 PDT	Summary: User timers broken for certain Dataflow streaming pipelines in multiple locations Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-08-18 21:30 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	19:44 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-08-18 21:30 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	15:57 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-08-18 20:00 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	10:49 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-08-18 16:00 US/Pacific. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	08:08 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: We are experiencing an issue with Google Cloud Dataflow beginning at Tuesday, 2022-08-09 08:35 US/Pacific. Missing output data due to Dataflow streaming pipelines using Runner v2 and setting user timers issue in the regions mentioned. in southamerica-west1 europe-west5 australia-southeast2 asia-south2 northamerica-northeast2 us-west3 us-east7 europe-central2 asia-northeast2 europe-west6 asia-east2 us-central2 southamerica-east1 asia-northeast3 asia-southeast2 us-west4 europe-north1 asia-south1 northamerica-northeast1 asia-east1 asia-northeast1 asia-southeast1 australia-southeast1 europe-southwest1 europe-west1 europe-west2 europe-west3 europe-west4 Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-08-18 11:00 US/Pacific with current details. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	08:03 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: We are experiencing an issue with Google Cloud Dataflow beginning at Tuesday, 2022-08-09 08:35 US/Pacific. Missing output data due to Dataflow streaming pipelines using Runner v2 and setting user timers issue in the regions mentioned in southamerica-west1 europe-west5 australia-southeast2 asia-south2 northamerica-northeast2 us-west3 us-east7 europe-central2 asia-northeast2 europe-west6 asia-east2 us-central2 southamerica-east1 asia-northeast3 asia-southeast2 us-west4 europe-north1 asia-south1 northamerica-northeast1 asia-east1 asia-northeast1 asia-southeast1 australia-southeast1 europe-southwest1 europe-west1 europe-west2 europe-west3 europe-west4 Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-08-18 09:00 US/Pacific with current details. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	07:39 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: We are experiencing an issue with Google Cloud Dataflow beginning at Tuesday, 2022-08-09 08:35 US/Pacific. Missing output data due to Dataflow streaming pipelines using Runner v2 and setting user timers issue in the regions mentioned. in southamerica-west1 europe-west5 australia-southeast2 asia-south2 northamerica-northeast2 us-west3 us-east7 europe-central2 asia-northeast2 europe-west6 asia-east2 us-central2 southamerica-east1 asia-northeast3 asia-southeast2 us-west4 europe-north1 asia-south1 northamerica-northeast1 asia-east1 asia-northeast1 asia-southeast1 australia-southeast1 europe-southwest1 europe-west1 europe-west2 europe-west3 europe-west4 Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-08-18 09:00 US/Pacific with current details. Diagnosis: Customers are only affected if they use a particular combination of Dataflow features: 1. they run in one of the affected regions above, 2. they run a streaming pipeline, not batch 3. they have enabled RunnerV2, which is the default for Python streaming, 4. they set user timers. There's no specific error message for this failure mode. The failure manifests in different ways, depending on the code of the pipeline. Workaround: Users can restart their streaming pipelines without RunnerV2, at the cost of having a small interruption of their pipeline.
18 Aug 2022	07:19 PDT	Summary: [multiregion-us] [multiregion-europe] [multiregion-sia] User timers broken for certain Dataflow streaming pipelines Description: We are experiencing an issue with Google Cloud Dataflow beginning at Tuesday, 2022-08-09 08:35 US/Pacific. Missing output data due to Dataflow streaming pipelines using Runner v2 and setting user timers issue in the regions mentioned. in southamerica-west1 europe-west5 australia-southeast2 asia-south2 northamerica-northeast2 us-west3 us-east7 europe-central2 asia-northeast2 europe-west6 asia-east2 us-central2 southamerica-east1 asia-northeast3 asia-southeast2 us-west4 europe-north1 asia-south1 northamerica-northeast1 asia-east1 asia-northeast1 asia-southeast1 australia-southeast1 europe-southwest1 europe-west1 europe-west2 europe-west3 europe-west4 Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-08-18 08:30 US/Pacific with current details. Diagnosis: Missing output data due to Dataflow streaming pipelines using Runner v2 and setting user timers issue in the regions mentioned. Workaround: None at this time.

All times are US/Pacific