Backend Monitoring Best Practices: SLOs, Golden Signals, Alerts, and Production Dashboards
A backend can return 200 OK and still be unhealthy for users. Requests can be technically successful but too slow, queues can fall hours behind, replicas can serve stale data, or a database can be minutes away from exhausting its connection pool.
Good monitoring starts from user-visible reliability and works inward toward the resources that explain it.
1. Define SLOs Before Collecting Thousands of Metrics
Start with questions users care about:
- Does checkout succeed?
- Is the API fast enough?
- Are background jobs completed within the promised time?
- Is data fresh enough?
Examples:
99.9% successful checkout requests over 30 days
p95 API latency below 400 ms
99% of invoice jobs complete within 2 minutes
An SLO gives monitoring a purpose. Without one, teams often alert on infrastructure noise rather than service reliability.
2. Put the Four Golden Signals on Every Critical Service
Google SRE's four golden signals remain an excellent baseline:
- Latency — request/job duration, including tail latency.
- Traffic — requests, jobs, messages, or another demand unit.
- Errors — failed or incorrect work.
- Saturation — how close the system is to a capacity limit.
For HTTP services, the RED shorthand is useful:
Rate
Errors
Duration
For infrastructure, USE is useful:
Utilization
Saturation
Errors
Do not stop at averages. Track p50, p95, and p99 for latency-sensitive paths.
3. Monitor Dependencies From the Caller’s Perspective
If an API depends on PostgreSQL, Redis, a payment provider, and a message broker, monitor the observed dependency behavior from the application as well as the dependency's own infrastructure.
Useful fields:
dependency latency
error rate
timeout rate
retry count
connection-pool usage
circuit-breaker rejections
A healthy database CPU graph does not prove the application is getting fast queries.
4. Database Monitoring Needs Workload Signals
For relational databases, useful signals include:
- connection pool saturation
- query latency
- slow-query count
- lock waits/deadlocks
- transaction duration
- replication lag
- cache/buffer hit behavior
- disk latency and storage growth
Do not page because one query was slow once. Alert when the behavior threatens an SLO or a capacity boundary.
5. Queues Need Depth and Age
Queue depth alone is ambiguous. A depth of 10,000 messages may be healthy if consumers clear them in seconds, and a depth of 100 may be terrible if the oldest item has waited an hour.
Track:
messages ready
messages in flight
oldest message age
processing rate
retry rate
DLQ growth
consumer errors
For asynchronous user promises, oldest message age is often the most meaningful reliability signal.
6. Monitor Resource Saturation Before It Reaches 100%
Watch resources that impose hard limits:
- CPU
- memory
- DB connections
- file descriptors
- worker concurrency
- thread/goroutine pools
- queue capacity
- disk space/IOPS
A system may degrade well before a resource reaches 100%. Learn thresholds from load tests and production behavior instead of picking arbitrary percentages.
7. Alerts Must Be Actionable
A page should mean someone should act now.
Bad alert:
CPU > 70% for 2 minutes
Better alert:
checkout error-budget burn is high
and p95 latency exceeds SLO
Use symptom-based paging for user impact and lower-priority notifications/dashboards for causes that do not yet require action.
Every serious alert should have:
- owner
- severity
- runbook
- dashboard link
- clear condition for resolution
8. Use Multi-Window Error-Budget Alerts
A brief spike and a slow multi-hour reliability burn are different incidents.
SLO-based alerting can combine fast and slow burn windows so teams catch both catastrophic failures and sustained degradation without paging on every tiny fluctuation.
The principle is more important than the exact formula: alert on how quickly reliability budget is being consumed.
9. Dashboards Should Support Decisions
A useful service dashboard normally answers:
- Are users affected?
- Which operation is affected?
- Is demand unusual?
- Which dependency is unhealthy?
- Is a resource saturated?
- Did a deployment/config change correlate with the problem?
Avoid dashboards with 80 graphs that nobody uses during an incident.
10. Correlate Metrics With Traces and Logs
Metrics tell you a problem is systemic. Traces tell you where individual requests spent time. Logs provide detailed event context.
Use OpenTelemetry or equivalent instrumentation to propagate trace context across HTTP, RPC, and queue boundaries.
11. Control Metric Cardinality
Never use unbounded labels such as:
user_id
request_id
session_id
raw URL containing UUID
These can create millions of time series and expensive telemetry pipelines.
Prefer bounded dimensions:
route=/orders/{id}
method=POST
status_code=500
region=ap-south
Put unique identifiers in traces/logs instead.
12. Treat Monitoring as a Production Dependency
Telemetry itself needs capacity controls:
- bounded buffers
- batching
- sampling
- retention
- ingestion monitoring
- backpressure
During an outage, log and trace volume often rises. Your monitoring pipeline should not collapse exactly when engineers need it most.
Production Checklist
- Critical user journeys have SLIs/SLOs
- Golden signals exist for every critical backend
- p95/p99 latency is visible
- Dependencies have caller-side latency/error telemetry
- Database pools, query latency, locks, and replication are monitored
- Queue depth and oldest message age are tracked
- Saturation limits are known
- Paging alerts are actionable and owned
- Dashboards show deployment/version dimensions
- Traces and logs correlate through trace IDs
- High-cardinality metric labels are avoided
- Monitoring pipeline health is monitored too
Final Takeaway
Backend monitoring should begin with what users expect, not with whatever metrics your infrastructure exports by default.
Define SLOs, instrument the golden signals, monitor dependency and queue behavior, track real saturation limits, and make alerts actionable. Then connect metrics to traces and structured logs so an alert can lead quickly to a cause.
The goal is not to collect more telemetry. It is to know when reliability is degrading, why it is happening, and what the team should do next.
References
BUILDING A SAAS PRODUCT?
Explore Lofingo’s end-to-end SaaS platforms, multi-tenant cloud architecture, and high-concurrency backend engineering.
EXPLORE SAAS SERVICES →