Service Health
Incident affecting Google Cloud SQL
Global: Cloud SQL PostgreSQL password validation errors with high connection load.
Incident began at 2022-02-15 09:14 and ended at 2022-02-19 10:46 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 23 Feb 2022 | 18:10 PST | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 15 February 2022 09:14 Incident End: 19 February 2022 10:46 Duration: 4 days, 1 hour, 32 minutes Affected Services and Features: CloudSQL- PostgreSQL connection errors Regions/Zones: Global Description: Applications using close to maximum number of connections, intermittently failed to connect to database instances. The connection error happened because Cloud SQL for PostgreSQL was using additional connections for internal use. Customer Impact: Customers impacted by this issue would have seen errors when establishing new connections to Cloud SQL for PostgreSQL instances. The error message would have been: FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already :: proc.c:347 Additional details: The issue was fully resolved on 19 February 2022 at 10:46 US/Pacific. In addition, the following prevention actions were completed:
|
| 22 Feb 2022 | 14:45 PST | The issue with Cloud SQL has been resolved for all affected instances as of Tuesday, 2022-02-22 14:30 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| 22 Feb 2022 | 11:22 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load. Description: Engineering team is currently working on mitigating the remainder of the affected instances (ETA EOB Feb 22, 2022). Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We will provide an update by Tuesday, 2022-02-22 19:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 18 Feb 2022 | 11:42 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load. Description: Engineering team is currently working on mitigating the non-HA instances (ETA Feb 22, 2022). The mitigation efforts are still underway for HA instances which continue to be affected. Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We will provide an update by Tuesday, 2022-02-22 12:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 17 Feb 2022 | 14:31 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load. Description: Engineering team continues to work on deploying the fix. Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We will provide an update by Friday, 2022-02-18 12:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 17 Feb 2022 | 11:30 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load. Description: Engineering team continues to work on deploying the fix. Affected customer instances have been mitigated and will not observe the password validation errors. We will provide an update by Thursday, 2022-02-17 17:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 16 Feb 2022 | 13:08 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load. Description: Affected customer instances have been mitigated and will not observe the password validation errors. Engineering team is currently working on rolling out a fix. ETA for rollout is unknown at this time. We will provide an update by Thursday, 2022-02-17 12:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 16 Feb 2022 | 10:11 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load Description: Engineering team is currently working on mitigation of the issue. We will provide an update by Wednesday, 2022-02-16 14:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 16 Feb 2022 | 06:22 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load Description: We believe the issue with FEATURE is partially resolved. We are currently performing checks to see if any instances have been missed. Full resolution is expected to complete by Wednesday, 2022-02-16 10:00 US/Pacific. We will provide an update by Wednesday, 2022-02-16 10:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 15 Feb 2022 | 19:28 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load Description: We believe the issue with Cloud SQL is partially resolved. Full resolution is expected to complete by Wednesday, 2022-02-16 06:30 US/Pacific. We will provide an update by Wednesday, 2022-02-16 06:30 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 15 Feb 2022 | 15:18 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load Description: Our engineering team continues to work on the mitigation of the issue. We will provide more information by Tuesday, 2022-02-15 19:30 US/Pacific. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection load. |
| 15 Feb 2022 | 12:44 PST | Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2022-02-15 15:30 US/Pacific. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection rate. |
| 15 Feb 2022 | 11:26 PST | Summary: Global: Cloud SQL Postgres instances experiences password validation error on some connections. Description: We are experiencing an issue with Cloud SQL beginning at Tuesday, 2022-01-26 11:24 US/Pacific. Our engineering team continues to investigate the issue, and after further investigation it was determined that the trigger conditions are not actually due to >50 QPS. We will provide an update by Tuesday, 2022-02-15 13:00 US/Pacific with current details. Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...". Workaround: Users can enable connection pooling to reduce the connection rate. |
| 15 Feb 2022 | 09:49 PST | Summary: Global: Some Cloud SQL Postgres instances have the following flag set and are unable to disable: cloudsql.enable_password_validation=on Description: All Postgres instances connections have been rate-limited to 50/sec to prevent triggering the issue. Customers may consider using the workaround to pool connections to avoid hitting the temporarily rate limit. We will provide more information by Tuesday, 2022-02-15 12:00 US/Pacific. Diagnosis: Affected postgres instances from a recent release have the following flag set and are unable to remove or disable this flag: cloudsql.enable_password_validation=on. This flag does not appear in Cloud Console, and attempting to disable flag via gcloud returns error where the flag is not recognized or supported. Password validation occurs on every new client connection but is limited to 50 QPS, and thus higher rates will return errors. Workaround: Users can enable connection pooling to avoid connecting more than 50 times a second. |
- All times are US/Pacific