A policy update containing unintended blank fields reached a code path in Service Control - the global layer that checks quota and policy on nearly every Google Cloud API call - that had a null-pointer defect and no feature-flag protection. Service Control crash-looped globally, returning 503s across dozens of products for roughly three hours; us-central1 took longest to recover due to a retry herd.
A race condition in the pipeline that propagates Google Cloud Load Balancer configuration caused a corrupt configuration to be pushed globally. Requests to sites fronted by GCLB began returning 404 errors even though the backends were healthy - taking down high-profile sites for roughly two hours until the configuration was rolled back.
A routine maintenance configuration change was applied to a far wider scope than intended, descheduling the network control-plane jobs in several US regions. The network fell back to preserving only high-priority traffic (like the control plane’s own signaling) while starving everything else, so Compute Engine, YouTube, Gmail, and Snapchat degraded severely for about four hours - worst in the US East.
Degradation in an internal blob-storage service that underpins Google Cloud Storage caused elevated error rates and latency for GCS operations, and for Google’s own products that store objects on the same layer - Gmail and Photos attachments among them - for roughly four hours.
6 min read
Recent incident log
Smaller incidents from the live feed (last 90 days). Major events graduate into full post-mortems above.
Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.
gcp