Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Console, Cloud Billing

Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Incident began at 2022-07-25 00:00 and ended at 2022-08-03 09:45 (all times are US/Pacific).

Previously affected location(s)

Global

Date Time Description
3 Aug 2022 15:20 PDT

Mini-IR (Preliminary IR)

We apologize for the inconvenience this incident may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 25 July 2021 00:00

Incident End: 03 Aug 2022 09:45

Duration: 9 days, 9 hours, 45 minutes

Affected Services and Features:

  • Google Cloud Console - Billing Export
  • Google Cloud Billing

Regions/Zones: Global

Description: Google Cloud Billing usage reporting experienced issues and may have shown incomplete data for a duration of 9 days, 9 hours, and 5 minutes. From preliminary analysis this was due to an issue with a recent rollout to the Google Compute Engine (GCE) control plane. The rollout produced unforeseen data change that caused a mismatch in Billing, and eventually a delay in the usage reporting.

Complete Usage data for all days in July are now available in the Cloud Billing Console and BigQuery Export. There was no impact to the invoices for July, which were issued on 1 August 2022.

Customer Impact:

Affected customers experienced the following:

  • Customers may have experienced delays in viewing real-time product usage information in the Cloud Billing Console or in BigQuery export, if used.
  • Cloud Billing usage reporting experienced issues and may have shown inaccurate or incomplete data.
    • Some customers may have seen up to double their instance usage being billed.
3 Aug 2022 09:53 PDT

This issue has been fully resolved as of 2022-08-03 09:45 US/Pacific.

  1. Complete Usage data for all days in July are now available in the Billing console, and BigQuery Export.
  2. There was no impact to the Invoices for July, which were issued on 2022-08-01.

This is the final update on this issue.

If you experience any further issues with Cloud Billing usage reporting or BigQuery export for the month of July, please contact Cloud Support.

We thank you for your patience while we worked on resolving the issue.

3 Aug 2022 09:42 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  1. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  2. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific.
  3. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 10:45 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 20:54 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  1. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  2. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific.
  3. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 20:52 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export. Data for 2022-07-29 is in process and expected to be fully available by mid-day Wednesday, 2022-08-03 US/Pacific. There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Wednesday, 2022-08-03 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 17:37 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  • Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  • Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
  • There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 17:06 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  • Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  • Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
  • There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 19:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 17:04 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  • Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  • Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
  • There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 12:55 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  • Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  • Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
  • There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

2 Aug 2022 09:01 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific.

  • Complete Usage data for all days, except 2022-07-29, are available, in the Billing console and Billing Export.
  • Data for 2022-07-29 is still under recovery and expected to be fully available by Tuesday, 2022-08-02 23:59 US/Pacific.
  • There is no impact to the Invoices for July, which were issued on 2022-08-01, and include accurate charge for 2022-07-29 as well.

We will provide more information by Tuesday, 2022-08-02 13:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

1 Aug 2022 21:16 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-25, 2022-07-26, and 2022-07-27 US/Pacific is now available. Over the next day, users will start to see usage data for remaining days with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Tuesday, 2022-08-02 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

1 Aug 2022 20:07 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Tuesday, 2022-08-02 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

1 Aug 2022 17:01 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Big Query Export will continue to be populated with usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Monday, 2022-08-01 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

1 Aug 2022 12:37 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific. Please note that the data in Billing Export to Big Query will also change for usage between 2022-07-25 and 2022-07-29 until the full recovery is completed.

We will provide more information by Monday, 2022-08-01 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

1 Aug 2022 08:46 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. Usage data for 2022-07-30 US/Pacific is now available. Over the next two days, users will start to see usage data from 2022-07-25 to 2022-07-29 become available with full recovery by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Monday, 2022-08-01 13:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 21:03 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Monday, 2022-08-01 09:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 20:45 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Sunday, 2022-07-31 21:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 17:49 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

Cloud Billing reporting has resumed normal operations from 2022-07-31 US/Pacific. We expect to have complete usage data from 2022-07-25 to 2022-07-30 available by Tuesday, 2022-08-02 23:59 US/Pacific.

We will provide more information by Sunday, 2022-07-31 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 17:20 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 19:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 14:55 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 12:07 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 15:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

31 Jul 2022 08:54 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

30 Jul 2022 19:59 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Sunday, 2022-07-31 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

30 Jul 2022 15:14 PDT

Summary: Cloud Billing usage reporting is experiencing issues, and Cost Management experience may show incomplete data.

Description: We are experiencing an issue with Cloud Billing. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 20:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information in the Cloud Billing Console.

Workaround: None at this time.

30 Jul 2022 09:01 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 14:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console.

Workaround: None at this time.

29 Jul 2022 20:55 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is still underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Saturday, 2022-07-30 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

29 Jul 2022 17:20 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and the recovery process is already underway.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Friday, 2022-07-29 21:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

29 Jul 2022 11:57 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs. The root cause has been identified, and we have now initiated the recovery process.

We do not have an ETA at this point for when the full recovery will be complete.

We will provide more information by Friday, 2022-07-29 17:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

29 Jul 2022 10:00 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Compute Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Google Compute Engine SKUs.

We have identified the root cause and mitigation is currently underway by our engineering team. However, we do not have an ETA for mitigation and resolution at this point.

We will provide more information by Friday, 2022-07-29 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

29 Jul 2022 08:29 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing where billing has been paused for a subset of Compute Engine SKUs.

We have identified the root cause and mitigation is currently underway by our engineering team. However, we do not have an ETA for mitigation and resolution at this point.

We will provide more information by Friday, 2022-07-29 12:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

29 Jul 2022 08:13 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-07-29 09:30 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.

28 Jul 2022 19:40 PDT

Summary: Cloud Billing is experiencing issues with usage reporting for some Google Cloud Engine SKUs

Description: We are experiencing an issue with Cloud Billing

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-07-29 09:00 US/Pacific.

Diagnosis: Users may experience delays in viewing real time Product usage information for some Google Compute Engine SKUs in the Cloud Billing Console

Workaround: None at this time.