gcpGlobal Google Cloud outage from an invalid automated policy update
Started
Jun 12, 2025, 5:49 PM UTC
Resolved
Jun 12, 2025, 9:00 PM UTC
Duration
3h 11m
Product
Cloud IAM
Event reference: inc-service-control-2025-0612
gcpdown.com independently monitors the public Google Cloud Service Health dashboard. How we source data.
Google Cloud impact
Workload type
Security & Identity
Impact
A malformed automated quota policy update to Service Control - the component that authorizes API requests - lacked error handling and hit a null pointer, crash-looping the binary and returning elevated 5xx errors across many Google Cloud services worldwide. Impact lasted just under three hours; us-central1 was worst-hit (~2h35m).
Affected product
Affected regions
Timeline
Updates newest first (times in UTC).
- Resolved
Request error rates have returned to normal across all regions. The issue is resolved and Google has published a post-incident review.
- Monitoring
A global mitigation switch has been deployed and error rates are recovering in most regions; us-central1 is draining a backlog and taking longer. We are monitoring recovery.
- Identified
The issue was traced to an invalid automated policy update in Service Control, which authorizes API requests. A global mitigation is being deployed.
- Investigating
We are investigating a global issue causing elevated error rates for multiple Google Cloud services.
Resolution
Request error rates have returned to normal across all regions. The issue is resolved and Google has published a post-incident review.