ELK Stack for Production Logging: Elasticsearch, Logstash, Kibana, Data Streams, and ILM

ELK Stack for Production Logging: Elasticsearch, Logstash, Kibana, Data Streams, and ILM

The ELK Stack originally referred to Elasticsearch, Logstash, and Kibana. Today the broader Elastic Stack includes additional collection and observability components, but the core logging architecture is still easy to understand:

The main architectural difference from label-oriented systems such as Loki is that Elasticsearch builds rich inverted indexes over fields, enabling powerful full-text search and filtering at the cost of more indexing and storage work.

What Each Component Does

Elasticsearch

Stores and indexes log documents so they can be searched, filtered, aggregated, and analyzed.

Logstash

A general-purpose data pipeline that can ingest from many sources, transform/enrich events, and route them to one or more destinations.

Kibana

Provides search, dashboards, visualizations, management, and observability workflows over Elasticsearch data.

Modern deployments may not need Logstash for every workload. Elastic Agent, OpenTelemetry, or Elasticsearch ingest pipelines can often handle simpler ingestion paths.

Use Data Streams for Append-Only Logs

Current Elastic documentation recommends data streams for continuously generated data such as logs and metrics.

A data stream gives applications one logical name while Elasticsearch manages multiple backing indices underneath:

logs-production-app
      ↓
index-000001
index-000002
index-000003

This makes rollover and lifecycle management much cleaner than manually creating daily indices.

Normalize Fields With ECS or OTel Conventions

If every application calls the same concept something different—host, hostname, serverName, machine—cross-service queries become painful.

Elastic Common Schema (ECS) standardizes common event fields for logs and metrics. Current Elastic integrations can also ingest OpenTelemetry-native logs and map between OTel and ECS-oriented representations in supported workflows.

Standard field naming improves:

• dashboards
• correlations
• reusable detections
• cross-service queries

Do not create a new schema vocabulary for every team.

Parse at Ingestion When the Structure Is Known

Elasticsearch ingest pipelines can run processors before indexing:

raw event
  ↓
parse timestamp
  ↓
extract fields
  ↓
rename / remove fields
  ↓
enrich
  ↓
index

Use Logstash when you need richer pipelines, many input/output plugins, complex transformations, or routing to multiple destinations.

For straightforward parsing and enrichment, an ingest pipeline can remove an extra moving part.

Mappings Matter

Elasticsearch determines how fields are indexed. Poor mappings create production problems.

Examples:

status_code → integer
@timestamp  → date
service.name → keyword
message      → text

Be careful with uncontrolled dynamic fields from arbitrary JSON payloads. A service generating thousands of unique field names can cause mapping explosion and cluster-state pressure.

Define templates for predictable datasets and avoid indexing fields you never search.

Full-Text Indexing Is Powerful—and Expensive

Elasticsearch is excellent when operators need queries such as:

find logs containing a specific error phrase
filter by service and region
aggregate errors by exception type
search arbitrary structured fields

But indexing every field has a cost in:

• CPU during ingestion
• disk
• memory
• merge work
• query complexity

Keep only useful searchable fields indexed. Large payloads that are retained only for reference may not need full indexing.

Lifecycle Management Controls Retention and Cost

Log data grows continuously.

Use lifecycle policies to automate rollover, retention, and storage transitions.

For self-managed/hosted Elasticsearch, Index Lifecycle Management (ILM) is a common mechanism. Elastic Cloud Serverless uses Data Stream Lifecycle (DLM) rather than ILM in several workflows.

A lifecycle policy should follow business requirements:

hot searchable logs → recent period
older searchable/archive tier → if needed
delete → after retention requirement

Do not retain every debug log forever simply because storage was cheap at the beginning.

Logging Levels Need Discipline

A production application should distinguish:

ERROR → operation failed
WARN  → abnormal but recoverable
INFO  → important state/business lifecycle
DEBUG → detailed diagnostic information

Always-on debug logging can multiply ingestion cost and bury important events.

Use temporary or targeted debug logging when possible.

Correlate Logs With Traces

Include safe correlation fields such as:

trace.id
transaction.id
service.name
service.version

Then an engineer can move from a slow distributed trace to the exact application logs produced during that request.

Logstash Should Be Used for a Reason

Logstash is valuable for:

• complex transformations
• multiple heterogeneous sources
• multiple output destinations
• plugin-heavy ingestion
• buffering patterns where configured appropriately

But it introduces another JVM-based service to size, monitor, secure, and scale.

Do not place Logstash between every producer and Elasticsearch if a lighter collector or ingest pipeline already meets the requirement.

Protect the Cluster From Ingestion Spikes

Logging traffic often spikes during failures—the exact moment you need the logging platform most.

Design for:

• bulk ingestion
• backpressure
• collector buffering
• bounded queues
• shard capacity
• disk headroom
• rejected-write monitoring

A logging outage during an application outage makes root-cause analysis much harder.

Shard Count Is an Architecture Decision

Too many tiny shards waste heap and cluster resources. Too few huge shards make recovery, relocation, and parallelism harder.

Use data streams and lifecycle/rollover policies to produce shard sizes appropriate for your workload rather than using “one index per service per day” blindly.

Benchmark with your ingestion rate, retention, query patterns, and node sizes.

Security and Multi-Tenancy

Protect log data because it often contains sensitive operational context.

Use:

• TLS
• authenticated ingest clients
• least-privilege API keys/roles
• index/data-stream permissions
• field/document controls where required
• network restrictions

Do not let clients decide unrestricted index names or query every tenant's logs.

Also redact secrets before ingestion. Once a password or token enters a replicated search cluster, cleanup becomes much harder.

ELK/Elastic vs Loki

Choose based on query needs.

Elastic is compelling when:

• arbitrary full-text search is core
• many structured fields must be indexed
• search/analytics features are heavily used
• the same platform supports security/search/observability use cases

Loki can be compelling when:

• queries start from bounded metadata labels
• object-storage economics matter
• full indexing of log content is unnecessary
• Grafana is the main operations UI

Neither is universally cheaper or better. Test real ingestion volume and operator query patterns.

Production Checklist

• Logs use data streams instead of uncontrolled manual index creation
• Field naming follows ECS/OTel conventions where practical
• Mappings/templates are deliberate
• Dynamic-field explosions are prevented
• Parsing happens in the simplest suitable layer
• Lifecycle/retention policy is configured
• Debug logging is controlled
• Trace IDs correlate logs and traces
• Ingestion spikes have buffering/backpressure
• Shard sizing is measured rather than guessed
• Ingest/query clients use least privilege
• Secrets and sensitive payloads are redacted

Final Takeaway

The ELK/Elastic stack is powerful because logs become richly searchable documents, not just files collected in one place.

Use that power deliberately. Standardize schemas, use data streams, manage retention, control mappings and shard growth, and avoid inserting Logstash where a simpler pipeline is enough. The strongest logging architecture is the one that gives engineers the searches they actually need without making ingestion cost, storage, or cluster operations dominate the application itself.

References

• Elastic: Log Monitoring
• Elastic: Data Streams
• Elastic: Ingest Pipelines
• Elastic: Logstash
• Elastic Common Schema