System Monitoring Explained: What to Measure, Alert On, and Investigate
Monitoring is the discipline of collecting and analyzing system signals so you can answer two basic questions:
- Is the service doing what users expect?
- If not, where should we investigate?
Black-Box vs White-Box Monitoring
Google SRE uses a useful distinction.
Black-box monitoring
Tests behavior from the outside, like a user would:
Can the homepage load?
Does checkout return successfully?
Does DNS resolve?
Is the API reachable from another region?
This is excellent for detecting real symptoms.
White-box monitoring
Uses internal telemetry:
CPU
DB connection usage
request latency
queue depth
cache hit rate
This helps explain causes and detect approaching capacity problems.
Strong monitoring uses both.
Start With the Four Golden Signals
Google SRE recommends:
- Latency — how long operations take.
- Traffic — demand on the service.
- Errors — failed/incorrect operations.
- Saturation — how close resources are to limits.
If you have no monitoring today, start here before collecting hundreds of specialized metrics.
Monitor User Journeys, Not Just Processes
A process being alive does not prove the product works.
A synthetic checkout can test:
For important flows, monitor what users experience from outside the service boundary.
Use SLOs to Define “Healthy”
Examples:
99.9% successful requests over 30 days
p95 latency below 500 ms
99% of jobs complete within 2 minutes
An SLO turns monitoring into a reliability contract rather than an arbitrary threshold collection.
Monitor Dependencies From Both Sides
If your backend says PostgreSQL calls are slow while PostgreSQL reports fast execution, the problem may be network or connection-pool wait time.
Measure caller-side:
dependency latency
errors
timeouts
retries
and dependency-side health where possible.
Watch Saturation Before Failure
Useful capacity signals include:
CPU
memory
disk space / IOPS
DB connections
queue age
worker concurrency
file descriptors
Many systems become slow before reaching a theoretical 100% limit. Learn practical thresholds from load tests and production behavior.
Alerts Should Be Actionable
A page should indicate urgent action.
Bad:
CPU crossed 70% once
Better:
Checkout error rate is consuming error budget rapidly
Google SRE emphasizes high-signal alerts and warns that noisy pages create alert fatigue.
Use dashboards/tickets for things that should be investigated but do not require immediate interruption.
Symptoms vs Causes
Symptom:
users are receiving HTTP 500s
Cause:
database rejects connections
Alert mostly on important symptoms. Use white-box telemetry to find the causes.
Keep Monitoring Simple
Remove alerts and metrics that nobody uses.
A monitoring system can become so complex that operators no longer understand why an alert fired.
Prefer a smaller set of predictable signals that have repeatedly proven useful.
Practical Monitoring Stack
A typical open-source design could be:
The products can change; the signal roles remain similar.
Production Checklist
- Important user journeys have black-box checks
- Golden signals exist for critical services
- SLOs define user-visible health
- Dependencies have latency/error/timeout telemetry
- Resource saturation is monitored
- p95/p99 latency is visible
- Alerts are urgent and actionable
- Noisy/unused alerts are periodically removed
- Dashboards make symptoms and causes easy to correlate
- Monitoring infrastructure itself is monitored
Final Takeaway
Monitoring should tell you when users are being affected and provide enough internal signals to start diagnosis.
Use black-box checks for external truth, white-box telemetry for causes, golden signals for service health, and SLOs to define reliability. Keep alerts simple and actionable.
You should never need someone to stare at a dashboard all day waiting for a graph to look strange.
References
BUILDING AN AI PRODUCT?
Talk to Lofingo about custom autonomous AI agents, enterprise workflow automation, and scalable software systems.
EXPLORE AI SERVICES →