gcp
Observability on Google Cloud has a trap built into it: the tooling you would reach for during an outage runs on the same platform that is having the outage. Cloud Monitoring, Cloud Logging, and your dashboards are excellent right up until a regional or global GCP event degrades them alongside your service. Good observability on GCP therefore has two halves: instrument deeply inside the platform, and keep at least one honest signal from outside it.
Start with the golden signals
Before SLOs, before fancy dashboards, capture the four golden signals for every critical service:
- Latency - how long requests take, split by successful and failed.
- Traffic - how much demand the service is handling.
- Errors - the rate of requests that fail, by type.
- Saturation - how full your most constrained resource is.
Cloud Monitoring collects these as metrics and Cloud Logging captures the detail behind them. Together they let you see the shape of a failure fast: a latency spike with rising saturation is a capacity problem; a clean error-rate jump with flat latency is often a dependency failure.
Define SLOs and error budgets
An SLO turns "is it up" into a number you can defend. Pick a few user-facing service level indicators, availability and latency are the usual pair, and set a target over a window:
| Term | Meaning | Who sets it |
|---|---|---|
| SLI | A measured indicator, e.g. percent of successful requests | You measure it |
| SLO | Your target for the SLI, e.g. 99.9 percent over 30 days | You set it |
| Error budget | The unreliability the SLO allows, e.g. 0.1 percent | Derived from the SLO |
| SLA | The provider's contractual promise, with credits if missed | The provider sets it |
The error budget is the useful part. At a 99.9 percent target you are allowed to fail one request in a thousand, and you get to decide how to spend that budget on incidents and risky releases. When the budget runs low, reliability work jumps the queue. Cloud Monitoring has native SLO tracking so you do not have to compute this by hand.
Alert on symptoms and burn rate
Alerting on every internal metric produces noise that trains people to ignore pages. Alert instead on what the user feels, and on how fast you are spending the error budget:
- Symptom alerts fire on user-visible failures: elevated error rate, latency past the SLO.
- Burn-rate alerts fire when you are consuming the error budget faster than the SLO allows. A fast burn pages someone now; a slow burn opens a ticket for the morning.
This two-tier approach means both a sudden outage and a slow reliability leak get the right response without paging the on-call for every transient blip.
The independent-check imperative
Here is the half that teams skip. If a GCP event takes down your service, it can take down your in-cloud monitoring at the same time, and you learn nothing from a blank dashboard. Run a synthetic probe from outside Google Cloud, from a different provider or a monitoring service, that hits your public endpoints on a schedule. When your Cloud Monitoring dashboards go dark, the external probe is the signal that tells you the truth and, crucially, tells you the outage is the platform's, not yours.
That distinction matters because the response differs completely. A regional GCP failure means execute your load balancing failover. A global one, like the June 2025 Service Control outage, means switch to graceful degradation, as covered in surviving Google Cloud outages. Independent detection is what lets you tell them apart in the first minute. Wire in outage alerts, watch the live GCP incident feed, and cross-reference historical provider data at clouddowntime.com/data.
Turn detection into recovery
An SLO breach caused by the provider is often an SLA breach too, which means service credits you are owed. Your monitoring already knows the exact impacted window, which is the hard part of any credit claim. cloudslacredit.com explains how that recovery works, and Next Signal turns your outage evidence into a ready-to-send claim and support ticket so the credit does not slip through the cracks after the incident review closes.