gcp
The single most useful thing you can do before designing for Google Cloud outages is to accept that there are two completely different kinds of them. One kind is scoped to a place: a zone or a region has a bad day, and the fix is to have capacity somewhere else. The other kind is scoped to a capability: a global product like load balancing or identity fails, and there is no somewhere else, because that product is the same everywhere at once. Most resilience mistakes come from treating the second kind like the first.
The two failure modes
Regional failures are the ones the textbooks are written for. A datacenter loses power, a region's network capacity collapses, a regional service has a bad deploy. The June 2019 US network congestion is the classic example: an entire region's network degraded and zone redundancy inside that region offered no escape. The answer is redundancy at a wider blast radius, which is what the multi-region architecture guide covers in depth.
Global failures are the ones that surprise well-architected teams. Google Cloud leans harder than other providers on global products, and that convenience carries a specific risk. The November 2021 GCLB outage and the June 2025 Service Control outage both hit customers who were spread across regions, because the broken component had no per-region isolation to begin with.
Map your dependencies first
Before you buy any redundancy, label every service you depend on:
| Scope | Examples | Escape route |
|---|---|---|
| Zonal | A single Compute Engine VM, a zonal disk | Spread across zones |
| Regional | Cloud SQL, GKE, single-region buckets | Add a second region |
| Global | Cloud Load Balancing, Cloud DNS, IAM, Service Control | None: degrade gracefully |
The table is the whole strategy in miniature. Anything in the top two rows can be made more resilient by duplicating it. Anything in the bottom row cannot, so you spend your effort on how the system behaves when that layer errors, not on trying to fail over from it.
Designing for regional failures
Zone redundancy is the non-negotiable baseline. Run compute across at least two zones, use regional persistent disks and regional managed instance groups, and put a regional load balancer in front. For Kubernetes specifically, follow the GKE high availability guide to get regional clusters and multi-zone node pools.
Add a second region only when you have written down a recovery time objective that a business owner has signed. Cross-region storage and geo-redundant backups are cheap and worth doing widely. Full active-active compute is expensive and worth doing rarely.
Designing for global failures
You cannot fail over from a global outage, so the design goal changes from "move traffic" to "keep serving something useful while the global layer is broken." Practical tactics:
- Cache identity and config. If IAM or a token service is unreachable, a short-lived cached credential or a cached policy decision keeps existing sessions alive instead of hard-failing.
- Fail static, not hard. When a global control plane returns errors, serve a cached or read-only version of your product rather than a blank error page.
- Retry with backoff and jitter. Global outages often present as elevated error rates rather than total loss. Client and service retries with jitter ride out a lot of transient global degradation.
- Decouple the critical path. If checkout depends on a global product, make sure the browsing and read paths do not, so a global failure degrades revenue instead of erasing the whole product.
Detect the failure before you fix it
Your recovery only starts when you know you are in an outage, and the provider status page is often minutes behind reality. Independent monitoring that watches your own endpoints tells you faster, and it tells you which kind of failure you are in, which decides your response. Wire in outage alerts and keep an eye on the live GCP incident feed on this site. Historical outage data across providers lives at clouddowntime.com/data if you want to size your risk with numbers.
When the outage was the provider's fault
If Google breached a published SLA, you are likely owed service credits, but they are never automatic. You have to detect the breach, calculate the impacted window, and file a claim. That is exactly the kind of recovery that gets forgotten in the scramble of an incident, so it helps to automate it. cloudslacredit.com explains how SLA credit recovery works, and Next Signal watches your cloud accounts and produces the claim and the support ticket for you. Surviving an outage is half the job; getting paid back for it is the other half.