Google Cloud Service Health

Google Cloud Service Health
Incidents
Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Cloud SQL

Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL

Incident began at 2022-08-29 09:32 and ended at 2022-08-29 16:16 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date	Time	Description
29 Aug 2022	16:16 PDT	The issue with Google Cloud SQL is believed to be affecting a very small number of projects and our Engineering Team is working on it. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We thank you for your patience while we're working on resolving the issue.
29 Aug 2022	15:03 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: Our engineering team continues to work on the mitigation of the issue. We will provide an update by Monday, 2022-08-29 17:30 US/Pacific with current details. Diagnosis: Customers may experience replication failure after maintenance on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: In most cases, internal automation should resolve the issue within 30 minutes. If internal automation does not resolve the issue, or if it is critical to restore service sooner, customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure. Please refer to documentation [1] [1] https://cloud.google.com/sql/docs/mysql/replication/manage-replicas
29 Aug 2022	13:24 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: Our engineering team continues to work on the mitigation of the issue. We will provide an update by Monday, 2022-08-29 15:30 US/Pacific with current details. Diagnosis: Customer may experience replication failure on the new CSA rollout on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: Customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure. Customers can use gcloud command to enable replication again on a failed replica server gcloud sql instances patch REPLICA_NAME --enable-database-replication
29 Aug 2022	12:44 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: Our engineering team continues to work on the mitigation of the issue. We will provide an update by Monday, 2022-08-29 13:30 US/Pacific with current details. Diagnosis: Customer may experience replication failure on the new CSA rollout on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: Customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure. Customers can use gcloud command to enable replication again on a failed replica server gcloud sql instances patch REPLICA_NAME --enable-database-replication
29 Aug 2022	11:22 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: Our engineering team is currently investigating the issue and working on mitigation steps. We will provide an update by Monday, 2022-08-29 13:00 US/Pacific with current details. Diagnosis: Customer may experience replication failure on the new CSA rollout on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: Customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure. Customers can use gcloud command to enable replication again on a failed replica server gcloud sql instances patch REPLICA_NAME --enable-database-replication
29 Aug 2022	10:55 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: Our engineering team is currently investigating the issue and working on mitigation steps. We will provide an update by Monday, 2022-08-29 12:00 US/Pacific with current details. Diagnosis: Customer may experience replication failure on the new CSA rollout on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: Customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure. Customers can use gcloud command to enable replication again on a failed replica server gcloud sql instances patch REPLICA_NAME --enable-database-replication
29 Aug 2022	09:47 PDT	Summary: Global: Conflict with timezone refresh and replica startup causes replication failure for Google Cloud SQL Description: We are experiencing an issue with Google Cloud SQL. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2022-08-29 11:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customer may experience replication failure on the new CSA rollout on replica, if the master instance is busy with INSERTs/UPDATEs on tables with timestamp column and session timezone is set to other than SYSTEM. Workaround: Customers can DISABLE REPLICA and ENABLE REPLICA on their replica instances to fix the replication failure.

All times are US/Pacific