Production Logging Best Practices: Structure, Security & Correlation

Production Logging Best Practices: Structured Logs, Correlation, Security, and Cost

Production logging is not console.log() with a bigger storage bill.

A useful log should tell an engineer what happened, where it happened, when it happened, and enough context to investigate it—without leaking secrets or turning every request into megabytes of noise.

Prefer Structured Events

Instead of:

Payment failed for order 912 because gateway timed out

prefer:

{
  "level": "error",
  "event": "payment_provider_timeout",
  "service": "payments",
  "order_id": "ord_912",
  "provider": "gateway_a",
  "duration_ms": 1200,
  "trace_id": "4fd2..."
}

Structured fields make logs easier to filter, aggregate, alert on, and correlate. OpenTelemetry's log model is designed around rich structured log records and correlation with trace/resource context.

Use Stable Event Names

Human messages can change; event names should remain stable.

event=payment_provider_timeout
event=order_created
event=login_denied

Dashboards and detections can depend on stable fields without parsing English sentences.

Log Levels Should Mean Something

A practical interpretation:

TRACE — extremely fine-grained diagnostic detail, usually disabled.
DEBUG — developer diagnostics.
INFO — meaningful normal lifecycle events.
WARN — abnormal/recoverable situation worth attention.
ERROR — an operation failed.
FATAL — process/system cannot continue safely.

Do not mark every caught exception as ERROR if it is an expected business outcome. And do not use INFO for every function entry/exit.

Add Request and Trace Correlation

In distributed systems, logs from one user action can appear in many services.

Rendering diagram…

Include trace_id and, where useful, span_id so logs can link directly to distributed traces. OpenTelemetry explicitly supports log correlation using execution context.

Include Service and Deployment Context

Useful bounded fields include:

service.name
service.version
environment
region
cluster
instance/pod

During incidents, error count by service.version is far more useful than trying to remember which deployment happened first.

Never Log Secrets

Do not log:

passwords
access/refresh tokens
API keys
session cookies
authorization headers
private encryption keys
raw payment card data
database credentials

OWASP's logging guidance also warns against recording data with stronger security classification than the log system is authorized to hold.

Minimize Personal and Sensitive Data

PII often enters logs accidentally through request bodies, query strings, error objects, or serialized user records.

Use data minimization:

Need full email? usually no
Need user correlation? perhaps internal pseudonymous ID/hash
Need full request body? rarely

OpenTelemetry's current security guidance recommends removing, hashing, filtering, or redacting sensitive attributes when necessary.

The cheapest sensitive log to protect is the one you never collected.

Sanitize User-Controlled Values

Attackers can place control characters or misleading content into inputs that later enter logs.

Treat log fields as data and use logging libraries/encoders rather than constructing raw lines manually. OWASP recommends sanitizing event data to prevent log injection and malformed records.

Separate Diagnostic and Audit Logging

They solve different problems.

Diagnostic logs

Help debug runtime behavior:

dependency timeout
job retry
cache miss spike

Audit/security events

Record important actions:

admin changed permissions
API key created
payment refunded
user login failed

Audit logs may need stronger integrity, access controls, and retention than ordinary debug logs.

Do Not Duplicate the Same Error at Every Layer

A common anti-pattern:

DB layer logs error
repository logs same error
service logs same error
controller logs same error

One failure becomes four nearly identical events.

Log where you can add meaningful context or where the error boundary is owned. Tracing can represent propagation without creating duplicate log spam.

Add Error Identity, Not Just Stack Traces

For failures, useful fields include:

error.type
error.code
operation
dependency
retryable

Stack traces are valuable for unexpected software faults but can be large and may contain sensitive values. Store them where appropriate and avoid turning them into indexed high-cardinality dimensions.

Keep Logging Off the Critical Path

A slow remote logging endpoint should not make checkout slow.

Prefer:

Rendering diagram…

Use buffering/batching and define what happens if the collector is unavailable. Avoid unbounded memory queues.

For critical audit events, durability requirements may be stronger than ordinary diagnostics and should be designed separately.

Control Debug Logging in Production

Debug logs can be invaluable during incidents and extremely expensive when permanently enabled.

Support controlled changes such as:

per-service temporary DEBUG
sampled detailed events
short TTL for elevated logging
approval/change tracking for sensitive systems

OWASP recommends ensuring security-required logging cannot simply be disabled and that logging-level changes follow appropriate controls.

Sampling Is Useful for High-Volume Repetitive Events

If a harmless success event occurs millions of times, retaining every copy may add little value.

Possible approaches:

sample repetitive debug/info events
retain all important errors/security events
aggregate numerical behavior as metrics instead

Do not sample away the events required for billing, auditing, compliance, or incident reconstruction.

Use Metrics for Numbers, Logs for Events

This is better:

metric: requests_total
metric: request_duration histogram

than emitting one log per request solely to calculate request rate and latency.

Logs should carry event context that metrics cannot efficiently represent.

Retention Should Match the Purpose

Different log classes need different lifetimes.

DEBUG logs       → short
normal app logs  → moderate
security/audit   → policy/compliance-driven

Keeping everything forever creates unnecessary cost and privacy risk. Deleting required evidence too early creates operational/compliance risk.

Define retention explicitly and apply it to backups/exports too.

Protect Logs as Sensitive Production Data

Centralized logs can expose internal architecture and business information.

Protect them with:

encryption in transit
encryption/storage controls at rest
least-privilege read access
access auditing
tamper protection for critical audit trails
tenant isolation

OWASP recommends protecting logs against unauthorized reading, modification, and deletion.

What Should a Normal Request Log Contain?

A useful HTTP completion event might include:

{
  "event": "http_request_completed",
  "method": "POST",
  "route": "/orders/{id}",
  "status_code": 201,
  "duration_ms": 84,
  "service": "orders",
  "trace_id": "abc..."
}

Notice the route template, not /orders/928381. Bounded route names are easier to aggregate and do not create unnecessary identifier leakage.

Production Checklist

Final Takeaway

Good logging is intentional telemetry, not a transcript of everything the application did.

Emit structured events with stable fields, preserve trace correlation, capture enough context to investigate failures, and aggressively avoid secrets and unnecessary personal data. Use metrics for aggregate numerical behavior and logs for the details that explain specific events.

The best log line is one that solves a real debugging, security, or operational question months later without creating a new privacy or cost problem today.

References

Tags:LoggingStructured LoggingOpenTelemetrySecurityObservabilityBackend EngineeringProduction Engineering

ENTERPRISE SAAS ENGINEERING

BUILDING A SAAS PRODUCT?

Explore Lofingo’s end-to-end SaaS platforms, multi-tenant cloud architecture, and high-concurrency backend engineering.

EXPLORE SAAS SERVICES →