Observability Explained: Logs, Metrics, Traces, OpenTelemetry, and SLOs
Monitoring tells you that something is wrong. Observability helps you investigate why.
In a distributed system, one request can cross a gateway, several services, a database, a cache, a queue, and an external API. A dashboard showing p95 latency = 2.8s is useful, but production debugging requires deeper questions: Which requests are slow? Which dependency dominates the trace? Did a deployment change the behavior? Is one region, tenant, or route affected?
Observability vs Monitoring
Monitoring usually answers known questions:
- Is CPU high?
- Is the error rate above threshold?
- Is disk nearly full?
- Is checkout violating its latency SLO?
Observability helps answer questions you did not know in advance:
- Why are only requests from one region slow?
- Why did latency rise while CPU stayed normal?
- Why does one tenant produce most DB lock waits?
- Which dependency caused the regression after release
v42?
A system is observable when its telemetry lets engineers move from a user-visible symptom to a useful explanation of the internal behavior that caused it.
The Three Core Signals
Logs
Logs record discrete events. Prefer structured fields over unstructured strings.
{
"service": "payments",
"level": "error",
"trace_id": "87f1...",
"order_id": "ord_921",
"event": "provider_timeout"
}
Good logs help answer what happened and preserve contextual details that would be too high-cardinality for metrics.
Do not log secrets, access tokens, raw payment data, or unnecessary personal information.
Metrics
Metrics summarize numeric behavior over time.
Useful examples:
http_requests_total
http_request_duration_seconds
queue_messages_pending
db_connections_active
cache_hit_ratio
Counters measure accumulating events, gauges measure current values, and histograms capture distributions such as request latency.
For latency, percentiles such as p50, p95, and p99 are usually more useful than one average.
Traces
A trace follows one operation across the system. Each unit of work is represented by a span.
This makes it obvious that the payment dependency—not PostgreSQL—is dominating latency.
Context Propagation Connects Distributed Work
Tracing works only if context follows the request across boundaries.
Propagate trace IDs across HTTP/RPC and message metadata. Without this, you get disconnected fragments instead of one useful distributed trace.
Why OpenTelemetry Matters
OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions, and the OpenTelemetry Collector for telemetry generation and transport.
The Collector can receive, batch, enrich, sample, and export telemetry without coupling every application directly to one observability vendor.
Instrument the important boundaries first:
- inbound/outbound HTTP
- gRPC
- SQL/database calls
- cache calls
- queue producers/consumers
- external APIs
- background jobs
Then add custom business spans and attributes where they improve diagnosis.
Cardinality: The Metric Cost Trap
Metric labels should come from bounded sets.
Good:
route=/orders/{id}
method=GET
status_code=200
region=ap-south
Dangerous:
user_id
request_id
session_id
full URL containing UUIDs
Unique IDs can create millions of time series and huge memory/storage costs. Put unique request identifiers in traces or logs instead.
The Four Golden Signals
Google SRE popularized four high-value signals:
- Latency — how long work takes.
- Traffic — demand on the system.
- Errors — failed or incorrect work.
- Saturation — how close the system is to capacity.
For request-driven services, the RED model—Rate, Errors, Duration—is also useful. For infrastructure resources, USE—Utilization, Saturation, Errors—can help organize dashboards.
Connect Telemetry to SLOs
An SLI is the measurement, such as successful checkout requests divided by total checkout requests.
An SLO is the target, for example:
99.9% of checkout requests succeed over 30 days
An SLA is a contractual commitment and may have business consequences.
Good observability prioritizes user-visible behavior rather than alerting on every internal metric.
Alert on Symptoms, Investigate Causes
Symptom:
checkout error rate is above SLO
Possible cause:
database connection pool exhausted
Paging on every possible cause creates alert fatigue. Prefer urgent alerts tied to user impact, then use traces, metrics, and logs to find the cause.
Sampling Controls Trace Cost
High-traffic systems often cannot retain every trace forever.
Head sampling
Decide early, before the trace completes.
sample 5% of traces
It is simple and cheap but may miss rare failures.
Tail sampling
Decide after enough of the trace is available.
This supports policies such as:
keep all errors
keep very slow traces
keep traces from new deployment
sample routine healthy traffic
Tail sampling is more powerful but requires more buffering and collector infrastructure.
The goal is not to sample aggressively. It is to retain the evidence most likely to matter during incidents.
Telemetry Needs Resource Limits Too
During incidents, error volume can spike exactly when telemetry is most valuable.
Protect the observability pipeline with:
- batching
- bounded queues
- memory limits
- sampling
- backpressure
- storage retention policies
An observability system that collapses during an outage is not production-ready.
A Practical Incident Workflow
The value comes from correlation. Metrics narrow the problem, traces reveal the path, and logs provide detailed event context.
Instrument Deployments and Versions
Many regressions begin after a change. Include safe dimensions such as:
service.version
release_sha
region
instance/pod
feature flag when relevant
Then you can query:
error rate by service.version
instead of relying on memory during an incident.
Observability Cost and Privacy Are Architecture Concerns
Telemetry can become expensive and sensitive.
Control:
- metric cardinality
- trace sampling
- log volume
- retention
- duplicated telemetry
- payload capture
And apply data minimization. Full request bodies, prompts, SQL parameters, or customer documents should not automatically become permanent observability data.
Production Checklist
- Logs are structured and privacy-aware
- Metrics include latency, traffic, errors, and saturation
- p95/p99 latency is visible where relevant
- Trace context propagates across services and queues
- OpenTelemetry or equivalent instrumentation covers key dependencies
- High-cardinality identifiers are kept out of metric labels
- Deploy/version metadata is queryable
- SLOs represent user-visible behavior
- Alerts focus on actionable symptoms
- Sampling keeps important error/slow traces
- Telemetry pipelines have backpressure and capacity limits
- Retention and sensitive-data policies are explicit
Final Takeaway
Observability is not “having Grafana” or “shipping logs.” It is the ability to explain production behavior from evidence.
Start with the golden signals. Instrument service and dependency boundaries. Propagate trace context everywhere work travels. Correlate logs, metrics, and traces. Tie alerts to SLOs. Control cardinality and sampling before telemetry cost becomes its own incident.
When an engineer can move from “users are slow” to “this release caused payment-provider calls in one region to queue for 570 ms” quickly and confidently, the system is becoming truly observable.
References
BUILDING AN AI PRODUCT?
Talk to Lofingo about custom autonomous AI agents, enterprise workflow automation, and scalable software systems.
EXPLORE AI SERVICES →