Service Health

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 15 February 2022 09:14

Incident End: 19 February 2022 10:46

Duration: 4 days, 1 hour, 32 minutes

Affected Services and Features:

CloudSQL- PostgreSQL connection errors

Regions/Zones: Global

Description:

Applications using close to maximum number of connections, intermittently failed to connect to database instances. The connection error happened because Cloud SQL for PostgreSQL was using additional connections for internal use.

Customer Impact:

Customers impacted by this issue would have seen errors when establishing new connections to Cloud SQL for PostgreSQL instances. The error message would have been: FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already :: proc.c:347

Additional details:

The issue was fully resolved on 19 February 2022 at 10:46 US/Pacific. In addition, the following prevention actions were completed:

• For impacted instances, Cloud SQL has disabled the use of additional connections.
• The maintenance rollout was canceled to prevent further instances from being affected.

The issue with Cloud SQL has been resolved for all affected instances as of Tuesday, 2022-02-22 14:30 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load.

Description: Engineering team is currently working on mitigating the remainder of the affected instances (ETA EOB Feb 22, 2022).

Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We will provide an update by Tuesday, 2022-02-22 19:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load.

Description: Engineering team is currently working on mitigating the non-HA instances (ETA Feb 22, 2022). The mitigation efforts are still underway for HA instances which continue to be affected.

Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We will provide an update by Tuesday, 2022-02-22 12:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load.

Description: Engineering team continues to work on deploying the fix.

Affected customer instances have been mitigated and will not observe the password validation errors. If you are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We will provide an update by Friday, 2022-02-18 12:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections to instances with high connection load - "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load.

Description: Engineering team continues to work on deploying the fix.

Affected customer instances have been mitigated and will not observe the password validation errors.

We will provide an update by Thursday, 2022-02-17 17:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load.

Description: Affected customer instances have been mitigated and will not observe the password validation errors.

Engineering team is currently working on rolling out a fix. ETA for rollout is unknown at this time.

We will provide an update by Thursday, 2022-02-17 12:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load

Description: Engineering team is currently working on mitigation of the issue.

We will provide an update by Wednesday, 2022-02-16 14:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load

Description: We believe the issue with FEATURE is partially resolved. We are currently performing checks to see if any instances have been missed.

Full resolution is expected to complete by Wednesday, 2022-02-16 10:00 US/Pacific.

We will provide an update by Wednesday, 2022-02-16 10:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load

Description: We believe the issue with Cloud SQL is partially resolved.

Full resolution is expected to complete by Wednesday, 2022-02-16 06:30 US/Pacific.

We will provide an update by Wednesday, 2022-02-16 06:30 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load

Description: Our engineering team continues to work on the mitigation of the issue.

We will provide more information by Tuesday, 2022-02-15 19:30 US/Pacific.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection load.

Summary: Global: Cloud SQL PostgreSQL password validation errors with high connection load

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2022-02-15 15:30 US/Pacific.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection rate.

Summary: Global: Cloud SQL Postgres instances experiences password validation error on some connections.

Description: We are experiencing an issue with Cloud SQL beginning at Tuesday, 2022-01-26 11:24 US/Pacific.

Our engineering team continues to investigate the issue, and after further investigation it was determined that the trigger conditions are not actually due to >50 QPS.

We will provide an update by Tuesday, 2022-02-15 13:00 US/Pacific with current details.

Diagnosis: Affected customers may see a password validation error message when establishing new connections: "FATAL: pwd_validation: Failed to connect to admin database, error: FATAL: sorry, too many clients already...".

Workaround: Users can enable connection pooling to reduce the connection rate.

Summary: Global: Some Cloud SQL Postgres instances have the following flag set and are unable to disable: cloudsql.enable_password_validation=on

Description: All Postgres instances connections have been rate-limited to 50/sec to prevent triggering the issue.

Customers may consider using the workaround to pool connections to avoid hitting the temporarily rate limit.

We will provide more information by Tuesday, 2022-02-15 12:00 US/Pacific.

Diagnosis: Affected postgres instances from a recent release have the following flag set and are unable to remove or disable this flag: cloudsql.enable_password_validation=on. This flag does not appear in Cloud Console, and attempting to disable flag via gcloud returns error where the flag is not recognized or supported. Password validation occurs on every new client connection but is limited to 50 QPS, and thus higher rates will return errors.

Workaround: Users can enable connection pooling to avoid connecting more than 50 times a second.