Google Cloud Service Health

Google Cloud Service Health
Incidents
Global: Issues with Cloud SQL for MySQL instance migration to 5.7 when source databases have gtid_mode set to ON.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Cloud SQL

Global: Issues with Cloud SQL for MySQL instance migration to 5.7 when source databases have gtid_mode set to ON.

Incident began at 2021-11-02 05:13 and ended at 2021-12-16 12:25 (all times are US/Pacific).

Date	Time	Description
20 Dec 2021	12:28 PST	We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 02 November 2021 04:13 Incident End: 16 December 2021 12:25 Duration: 44 days, 8 hours, 12 minutes Affected Services and Features: CloudSQL- MySQL instance migration. Regions/Zones: Global Description: CloudSQL experienced an issue with MySQL instance migration to 5.7.36 with source database gtid_mode set to ON for a duration of 44 days, 8 hours, 12 minutes. The root cause of the issue is a MySQL bug https://bugs.mysql.com/bug.php?id=105761 that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021, from a source database that had gtid_mode set to ON. Customer Impact: Cloud SQL External Server or Database Migration Service (DMS) users who migrated to Cloud SQL for MySQL 5.7.36, with replication using managed dumps may have experienced replication failures if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those jobs during the replication phase, the migration job may have failed. The issue was observed when ALL the following conditions are met: Cloud SQL MySQL version migrated to is 5.7.36. Source database gtid_mode is ON. Customer is running continuous migration (see DMS doc https://cloud.google.com/database-migration/docs/mysql/create-migration-job). Customer is not migrating with a file, i.e. migration is using a managed dump. New instances should not experience this issue, but please follow the below recommended workaround for existing instances that are impacted. We will be sending out targeted communications to the affected customers. Workaround: Customers should start a new migration by dumping the data from a source database instance running MySQL 5.7.35 or lower versions to Google Cloud Storage (GCS) manually (Cloud SQL, DMS), using a mysqldump file, and then migrate using the dump file to Cloud SQL for MySQL 5.7.
16 Dec 2021	13:13 PST	The issue with Cloud SQL has been resolved as of Thursday, 2021-12-16 12:25 US/Pacific. New instances will no longer experience this issue, but please follow the recommended workaround for existing instances that are impacted. We will be sending out targeted communications to the affected customers. We thank you for your patience while we worked on resolving the issue.
13 Dec 2021	14:00 PST	Summary: Global: Issues with Cloud SQL for MySQL instance migration to 5.7 when source databases have gtid_mode set to ON. Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete over the next few days. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL for MySQL 5.7 with GTID replication using managed dumps may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those jobs during the replication phase, the migration job may fail. Workaround: Customers should start a new migration by dumping the data from a source database instance running MySQL 5.7.35 or lower versions to Google Cloud Storage (GCS) manually (Cloud SQL, DMS), using a mysqldump file, and then migrate using the dump file to Cloud SQL for MySQL 5.7.
8 Dec 2021	22:04 PST	Summary: Global: Issues with Cloud SQL for MySQL instance migration to 5.7 when source databases have gtid_mode set to ON. Description: Our engineering team continues to investigate the issue caused by a MySQL bug https://bugs.mysql.com/bug.php?id=105761 that may affect customers who migrated to MySQL 5.7.36 on or after November 2nd 2021, from a source database that had gtid_mode set to ON. The issue is being observed when ALL of the following conditions are met: Cloud SQL for MySQL version migrated to is 5.7.36 source database gtid_mode is ON The migration was a continuous migration (see DMS doc https://cloud.google.com/database-migration/docs/mysql/configure-source-database) The migration was performed using an auto-generated initial dump, also called a managed dump. We will provide further updates by Monday, 2021-12-13 17:00 US/Pacific with details. In the meantime, we recommend halting any migrations to Cloud SQL for MySQL 5.7 where the source database has gtid_mode set to ON and is using managed dumps, to prevent this scenario from occurring. The workaround outlined below may be used for these migrations instead. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL for MySQL 5.7 with GTID replication using managed dumps may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those jobs during the replication phase, the migration job may fail. Workaround: Customers should start a new migration by dumping the data from a source database instance running MySQL 5.7.35 or lower versions to Google Cloud Storage (GCS) manually (Cloud SQL, DMS), using a mysqldump file, and then migrate using the dump file to Cloud SQL for MySQL 5.7.
8 Dec 2021	17:21 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36 from source databases that have gtid_mode set to ON. Description: Our engineering team continues to investigate the issue caused by a MySQL bug https://bugs.mysql.com/bug.php?id=105761 that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021, from a source database that had gtid_mode set to ON. The issue is being observed when ALL of the following conditions are met: Cloud SQL MySQL version migrated to is 5.7.36 source database gtid_mode is ON Customer is running continuous migration (see DMS doc https://cloud.google.com/database-migration/docs/postgres/create-migration-job) Customer is not migrating with a file, i.e. migration is using a managed dump. We will provide further updates by Monday, 2021-12-13 17:00 US/Pacific with details. In the meantime, we recommend halting any migrations to Cloud SQL MySQL version 5.7 where the source database has gtid_mode set to ON to prevent this scenario from occurring. The workaround outlined below may be used for these migrations. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those jobs during the replication phase, the migration job may fail. Workaround: Customers should start a new migration by dumping the data from source MySQL 5.7.35 or lower versions to Google Cloud Storage (GCS) manually (Cloud SQL, DMS), using mysqldump, and then migrate using the file to Cloud SQL MySQL 5.7 version.
8 Dec 2021	13:59 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36. Description: Our engineering team continues to investigate the issue that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021 We will provide an update by Wednesday, 2021-12-08 17:00 US/Pacific with current details. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those data during replication phase, the migration job may fail. Workaround: Customers may need to start a new migration by dumping the data from source mysql 5.7.35 or lower versions to Google Cloud Storage (GCS) manually, using mysqldump, and then migrate the file to destination mysql version.
7 Dec 2021	22:28 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36. Description: Our engineering team continues to investigate the issue that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021 We will provide an update by Wednesday, 2021-12-08 14:00 US/Pacific with current details. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those data during replication phase, the migration job may fail. Workaround: Customers may need to start a new migration by dumping the data from source mysql 5.7.35 or lower versions to Google Cloud Storage (GCS) manually, using mysqldump, and then migrate the file to destination mysql version.
7 Dec 2021	19:31 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36. Description: Our engineering team continues to investigate the issue that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021 We will provide an update by Tuesday, 2021-12-07 23:30 US/Pacific with current details. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those data during replication phase, the migration job may fail. Workaround: Customers may need to start a new migration by dumping the data from source mysql 5.7.35 or lower versions to Google Cloud Storage (GCS) manually, using mysqldump, and then migrate the file to destination mysql version.
7 Dec 2021	16:35 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36. Description: Our engineering team continues to investigate the issue that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021 We will provide an update by Tuesday, 2021-12-07 19:30 US/Pacific with current details. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those data during replication phase, the migration job may fail. Workaround: Customers may need to start a new migration by dumping the data from source mysql 5.7.35 or lower versions to Google Cloud Storage (GCS) manually, using mysqldump, and then migrate the file to destination mysql version.
7 Dec 2021	14:36 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36 Description: Our engineering team continues to investigate the issue that may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021. We will provide an update by Tuesday, 2021-12-07 16:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who migrate to Cloud SQL MySQL 5.7 with GTID replication may experience errors if they have insert/update/delete transactions during the dump phase. Furthermore, if updates were made to those data during replication phase, the migration job may fail. Workaround: Customers need to start a new migration by dumping the data from source mysql 5.7.35 or lower versions to Google Cloud Storage (GCS) manually, using mysqldump, and then migrate the file to destination mysql version.
7 Dec 2021	13:14 PST	Summary: Global: Issues with Cloud SQL migrating to MySQL version 5.7.36 Description: We are experiencing an issue with Cloud SQL when migrating to MySQL version 5.7.36. The issue may affect customers who migrated to MySQL version 5.7.36 on or after November 2nd 2021 Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-12-07 14:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Cloud SQL External Server or Database Migration Service (DMS) users who want to migrate to Cloud SQL MySQL 5.7 with GTID replication may experience data inconsistency (i.e. data in Cloud SQL may not be consistent with data on the source) issue if they have insert/update/delete transactions during the dump phase. Further more, if customer has updates to those inconsistent data during replication phase, the migration job may fail. Workaround: Customers needs to start a new migration by dumping the data from source by themselves to Google Cloud Storage (GCS) and then migrate with the file.

All times are US/Pacific