The November 2021 Google Cloud Load Balancer Outage: A Config Race Condition

Root cause

Google Cloud Load Balancing is a global product - a single anycast front end distributes traffic to backends anywhere in the world - which is exactly what makes it powerful and what made this incident global. The configuration that tells the load balancer how to route requests is generated and propagated through a pipeline, and on November 16 a race condition in that pipeline produced a corrupt configuration that was pushed to the fleet. With bad routing config, the load balancer couldn't match incoming requests to their backend services and returned 404 Not Found - for traffic it never forwarded to origins that were perfectly healthy.

This is the signature failure mode of global control planes: the data plane (your servers) is fine, but the global layer that fronts it has failed, so from the user's perspective everything is down. There is nothing you can do at the origin to fix a load-balancer config problem.

Business impact

Because GCLB fronts a large share of the web's high-traffic properties, the casualty list read like a consumer-internet roll call: Spotify, Home Depot, Snap, and many more served 404s simultaneously. For each of them, dashboards showed healthy backends and zero origin errors - because the requests were dying at Google's edge, never reaching the origin. That disconnect makes these incidents especially disorienting to diagnose from inside the affected company.

Prevention and lessons

1. Config changes need the same rigor as code deploys. A race condition in a propagation pipeline is a software defect in disguise. Staged rollout, validation of generated config before push, and fast automated rollback are the controls that turn a two-hour global event into a one-region blip.
2. Understand what "global product" means for your blast radius. GCLB, Cloud DNS, and Service Control are global by design - being multi-region does not protect you from them. Our multi-region GCP guide covers which products are global and how to reason about the coupling.
3. Have an out-of-band health signal. When your own dashboards show green backends but users report outages, an external check (outage alerts from outside your infrastructure) is what tells you the problem is the edge, not you.
4. Watch Cloud Load Balancing status directly during "everything is 404ing but our servers are fine" incidents - it's the fastest way to confirm the fault is Google's edge.