gcpRoot cause
Google Cloud, like the other hyperscalers, runs a shared layer that sits in front of nearly every API call - Service Control, which enforces quota and checks policy before a request reaches the product behind it. On June 12, a change to quota-policy data was inserted that contained unintended blank fields. When Service Control read that data, it exercised a code path with a null-pointer defect - and, critically, that path was not protected by a feature flag, so there was no gradual rollout to catch the crash before it went global. Service Control began crash-looping in every region at once.
Two properties made this a global event rather than a contained one. First, Service Control is
in the request path for almost everything, so its failure isn't a degraded product - it's a
503 on the front door of the whole platform. Second, the missing feature flag meant the bad data
reached every region simultaneously; a staged rollout would have failed one region and been rolled
back.
Business impact
The blast radius was enormous and shallow: dozens of Google Cloud products returned errors while the products themselves were fundamentally healthy - they simply couldn't be reached through the API layer. The visible casualties told the story of how much of the internet sits on Google Cloud: Cloudflare's Workers KV (which depended on Google Cloud), Spotify, and Discord all degraded during the window. For individual customers, anything that made Google Cloud API calls on its hot path - which is to say almost everything - saw errors until Service Control recovered.
The recovery also demonstrated the retry herd problem: when the mitigation landed, every client that had been failing for an hour reconnected at once, and us-central1's Service Control tasks - the largest fleet - buckled under the reconnection storm and took longest to stabilize.
Prevention and lessons
- Feature-flag every code path that reads external data. The defect existed for a while harmlessly; the outage happened because the triggering data reached it globally with no staged rollout. Flags that fail one region first are the single highest-leverage safety control.
- Design for control-plane independence. Workloads that cached credentials and results, and that didn't call the API server on every request, degraded less. The GKE high availability guide covers keeping your data plane serving when the control/API plane is down - the defining resilience skill for GCP.
- Bound your retries. The us-central1 tail was a self-inflicted retry herd. Exponential backoff with jitter protects both you and the recovering shared service.
- Know your global dependencies. "We're multi-region" is no defense against a global-layer failure. Map which global products (Service Control, Cloud Load Balancing, Cloud DNS) your architecture depends on - see multi-region GCP patterns.