gcpNetwork congestion across multiple US regions
Started
Jun 2, 2019, 6:45 PM UTC
Resolved
Jun 2, 2019, 11:00 PM UTC
Duration
4h 15m
Product
Virtual Private Cloud
Event reference: inc-networking-2019-0602
gcpdown.com independently monitors the public Google Cloud Service Health dashboard. How we source data.
Google Cloud impact
Workload type
Networking
Impact
A configuration change intended for a small number of servers in one region was incorrectly applied to a much larger set across several US regions, dropping more than half of their network capacity and causing severe congestion in parts of us-east, us-central, and us-west for about four hours.
Affected product
Affected regions
Timeline
Updates newest first (times in UTC).
- Resolved
Full network capacity has been restored and services are operating normally. The issue is resolved.
- Monitoring
Network capacity is being returned to service and congestion is easing. We are monitoring as traffic normalizes.
- Identified
A configuration change meant for a small group of servers was applied more broadly by automation, removing over half the network capacity in several US regions. Engineers are restoring capacity.
- Investigating
We are investigating high levels of network congestion affecting services in the United States.
Resolution
Full network capacity has been restored and services are operating normally. The issue is resolved.