gcpGlobal 404 errors for external HTTP(S) Load Balancing
Started
Nov 16, 2021, 5:35 PM UTC
Resolved
Nov 16, 2021, 7:28 PM UTC
Duration
1h 53m
Product
Cloud Load Balancing
Event reference: inc-load-balancing-2021-1116
gcpdown.com independently monitors the public Google Cloud Service Health dashboard. How we source data.
Google Cloud impact
Workload type
Networking
Impact
A bug in the configuration pipeline that propagates rules to Google’s external HTTP(S) Load Balancing (GCLB) caused affected sites to return 404 errors, with knock-on impact to App Engine, Cloud Functions, Cloud Run, and Apigee. Total impact was about 1 hour 53 minutes; a rollback to the last-known-good config resolved it.
Affected product
Affected regions
Timeline
Updates newest first (times in UTC).
- Resolved
The rollback completed and 404 errors for external HTTP(S) Load Balancing have cleared globally. The issue is resolved.
- Identified
The errors were caused by a bug in the configuration pipeline that pushes rules to GCLB. We are rolling back to the last known good configuration.
- Investigating
We are investigating 404 errors for services served through external HTTP(S) Load Balancing.
Resolution
The rollback completed and 404 errors for external HTTP(S) Load Balancing have cleared globally. The issue is resolved.