Incident Report

Summary:

On Monday, 25 July 2022, a problem with Google Cloud Billing resulted in incomplete reporting data in the Billing Console for a duration of 9 days, 9 hours and 5 minutes. Invoices were not impacted during the outage. To our customers whose operations were impacted during this outage, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause:

Google periodically rolls out updates to its products to improve performance and reliability, and to add new functionality. On Monday, 25 July 2022 07:08 US/Pacific, a rollout to Google Compute Engine (GCE) resulted in a change to usage data which created a mismatch in customer billing information and eventually delayed reporting.

Remediation and Prevention:

Google engineers were alerted to the delay in usage reporting through automated monitoring on Tuesday, 26 July 2022 at 01:10 US/Pacific and immediately started an investigation. Google engineers began rolling out a fix on Tuesday, 26 July 2022 at 20:29. However, during the mitigation engineers identified a second issue, which manifested as duplicated results and overbilled charges displayed for 25 July, 26 July, and 27 July.

At 15:27 on Wednesday, 27 July 2022, having found the root cause of the problem Google engineers suspended the relevant GCE rollout and developed a fix to correct the duplicated results. On Thursday, 28 July 2022, Google engineers continued to work on mitigation efforts and completed a comprehensive manual verification of the fix before pushing it to production. On Friday, 29 July 2022, engineers resumed the billing system to reprocess the customer usage data. In parallel, engineers observed a failure in billing reporting due to the large amount of queued corrections.

The incident was fully mitigated on Wednesday, 03 August 2022 09:47 US/Pacific, once the correction process completed across all systems.

There was no impact to July invoices, which were issued on Monday, 01 August 2022.

Google is committed to preventing future issues like this and is completing the following actions to ensure there is not a recurrence:

• Implement validation checks to detect the duplicated results that were passed to downstream systems, causing the second issue.
• Implement end-to-end testing to cover critical user journeys across GCE, Billing, and Invoicing.
• Implement a change detection program to identify backward incompatible events from GCE to Billing to prevent transient incorrect billing results.
• Update the GCE rollout procedure to minimize impact when rolling out a new global billing capability for GCE.

We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact:

Google Compute Engine (GCE), including Google Kubernetes Engine (GKE) and Google App Engine Flex (GAE):

• 100% of GCE, GAE Flex, and GKE customers may have experienced delays in viewing real-time product usage information in the Cloud Billing Console or in BigQuery export, if used.
• Cloud Billing usage reporting experienced issues and may have shown inaccurate or incomplete data.
  • Some customers may have seen up to double their instance usage being billed.

Mini-IR (Preliminary IR)

We apologize for the inconvenience this incident may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 25 July 2021 00:00

Incident End: 03 Aug 2022 09:45

Duration: 9 days, 9 hours, 45 minutes

Affected Services and Features:

• Google Cloud Console - Billing Export
• Google Cloud Billing

Regions/Zones: Global

Description: Google Cloud Billing usage reporting experienced issues and may have shown incomplete data for a duration of 9 days, 9 hours, and 5 minutes. From preliminary analysis this was due to an issue with a recent rollout to the Google Compute Engine (GCE) control plane. The rollout produced unforeseen data change that caused a mismatch in Billing, and eventually a delay in the usage reporting.

Complete Usage data for all days in July are now available in the Cloud Billing Console and BigQuery Export. There was no impact to the invoices for July, which were issued on 1 August 2022.

Customer Impact:

Affected customers experienced the following:

• Customers may have experienced delays in viewing real-time product usage information in the Cloud Billing Console or in BigQuery export, if used.
• Cloud Billing usage reporting experienced issues and may have shown inaccurate or incomplete data.
  • Some customers may have seen up to double their instance usage being billed.

This issue has been fully resolved as of 2022-08-03 09:45 US/Pacific.

1. Complete Usage data for all days in July are now available in the Billing console, and BigQuery Export.
2. There was no impact to the Invoices for July, which were issued on 2022-08-01.

This is the final update on this issue.

If you experience any further issues with Cloud Billing usage reporting or BigQuery export for the month of July, please contact Cloud Support.

We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

1. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
2. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific.
3. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 10:45 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

1. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
2. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific.
3. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

• Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
• Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
• There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

• Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
• Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
• There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 19:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

• Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
• Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
• There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

• Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
• Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
• There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

• Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
• Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
• There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 13:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-25, 2022-07-26, and 2022-07-27 US/Pacific is now available. Over the next day, users will start to see usage data for remaining days with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Tuesday, 2022-08-02 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Tuesday, 2022-08-02 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Monday, 2022-08-01 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Billing Export to Big Query will also change for usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Monday, 2022-08-01 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Monday, 2022-08-01 13:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Monday, 2022-08-01 09:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Sunday, 2022-07-31 21:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Sunday, 2022-07-31 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 19:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 15:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 20:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 14:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console.

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is already underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Friday, 2022-07-29 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and we have now initiated the recovery process.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Friday, 2022-07-29 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Google Compute Engine SKUs.

We have identified the root cause and mitigation is currently underway by our engineering team. However, we do not have an ETA for mitigation and resolution at this point.

We will provide more information by Friday, 2022-07-29 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs.

We have identified the root cause and mitigation is currently underway by our engineering team. However, we do not have an ETA for mitigation and resolution at this point.

We will provide more information by Friday, 2022-07-29 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-07-29 09:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-07-29 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.