Grafana Loki in Production: LogQL, Labels, OTLP, Storage, and Multi-Tenancy
Grafana Loki is a horizontally scalable, multi-tenant log aggregation system designed around a different indexing model from traditional full-text log engines: it indexes metadata labels, while compressed log content is stored in chunks.
That design can keep logging infrastructure simpler and cheaper—but only if labels, storage, retention, and query patterns are designed carefully.
Labels Are the Core Index
A Loki stream is identified by a set of labels, for example:
{service="payments", environment="production", region="ap-south"}
Labels should describe bounded, frequently useful dimensions.
Good labels:
service
environment
cluster
namespace
region
Dangerous labels:
request_id
user_id
trace_id
pod UID
unique URL
High-cardinality labels create huge numbers of streams and hurt ingestion/query efficiency.
Structured Metadata Solves the High-Cardinality Problem
Current Loki supports structured metadata for useful fields that should travel with a log line but should not become indexed labels.
Examples:
trace_id
process_id
pod name
thread_id
user-specific identifiers where policy permits
Grafana's current documentation notes that structured metadata requires the TSDB index and schema v13+, and is central to native OTLP log ingestion.
A practical rule:
> Put low-cardinality routing/search dimensions in labels; put high-cardinality context in structured metadata or the log body.
Native OTLP Ingestion
Modern OpenTelemetry pipelines can send logs to Loki through OTLP.
This makes it easier to use one telemetry instrumentation ecosystem for logs, metrics, and traces while routing each signal to an appropriate backend.
Keep an eye on which OTel resource attributes become Loki labels. Promoting high-cardinality attributes into indexed labels can recreate the same cardinality problem through automatic configuration.
LogQL: Query Logs and Derive Metrics
Loki uses LogQL.
A simple selector:
{service="payments", environment="production"}
Filter matching lines:
{service="payments"} |= "timeout"
Parse structured logs:
{service="payments"} | json | level="error"
LogQL can also derive metric-style values from logs, such as error rates over time.
This is powerful, but a log query should not replace purpose-built metrics for every dashboard. Metrics are usually more efficient for high-frequency numerical monitoring.
Current Storage Baseline: TSDB + Object Storage
Grafana's current Loki documentation recommends the TSDB index with schema v13 for modern deployments. Log chunks can live in object storage such as S3, GCS, or Azure-compatible storage.
Conceptually:
small searchable metadata index
+
compressed log chunks in object storage
This separation is one reason Loki can scale economically for large log volumes.
Legacy index types should not be chosen for new deployments unless there is a specific compatibility requirement.
Retention Is Not Automatic by Default
Current Loki retention is handled by the Compactor.
Operators should define retention deliberately:
limits_config:
retention_period: 744h
Retention can be configured globally, per tenant, and for streams selected by labels.
Important operational point: object-store lifecycle policies should not delete chunks earlier than Loki's configured retention process expects, or queries can encounter missing data.
Multi-Tenancy Must Be Enforced at the Trusted Boundary
Loki is designed as a multi-tenant system. Requests identify a tenant using the X-Scope-OrgID header in multi-tenant mode.
Never let an untrusted client freely choose that header and talk directly to Loki.
Prefer:
Authentication/authorization belongs in the gateway or surrounding platform; Loki tenant IDs should be derived from trusted identity.
Set Per-Tenant Limits
A noisy customer or broken service can produce enormous log volume.
Use limits for:
- ingestion rate
- burst size
- stream count
- query parallelism
- query range
- structured metadata size
- retention
Multi-tenancy without resource isolation can still allow one tenant to degrade shared infrastructure.
Keep Log Lines Structured
Even when Loki does not fully index log bodies, structured JSON/logfmt makes parsing cheaper and queries clearer.
Prefer:
{"level":"error","event":"payment_timeout","provider":"bank_a","duration_ms":920}
instead of:
Something went wrong while payment was happening...
Do not log secrets, access tokens, passwords, authorization headers, or full sensitive payloads.
Link Logs and Traces
Store a trace_id as structured metadata or safe log context so Grafana can correlate a log line with the distributed trace that caused it.
That correlation is far more useful than maintaining three isolated observability tools.
Query Performance Starts at Ingestion
Loki query performance is strongly influenced by stream selection.
A query that begins with useful labels:
{service="orders", environment="production"}
can narrow the dataset before parsing lines.
A query that effectively asks Loki to scan huge time ranges across every service and run expensive regex parsing will cost much more.
Design labels around real operational query patterns—but resist indexing high-cardinality values just to make one query convenient.
Scale the Components That Are Actually Saturated
Distributed Loki separates ingestion and query responsibilities. At scale, components such as distributors, ingesters, queriers, query frontends, and compactors have different resource profiles.
Monitor:
- ingestion bytes/sec
- rejected/discarded samples
- active streams
- ingester memory
- object-storage latency/errors
- query latency
- query queue depth
- compactor health
- cache effectiveness where configured
Do not scale the entire deployment uniformly if only query capacity is saturated.
Loki vs Elasticsearch/OpenSearch for Logs
Loki is attractive when:
- logs are commonly filtered first by known metadata labels
- Grafana is already central to operations
- object-storage economics matter
- full arbitrary text indexing of every token is unnecessary
A full-text search engine can be better when deep arbitrary textual search and rich inverted indexing across most fields are core requirements.
The right choice depends on query behavior, not logo preference.
Production Checklist
- Modern deployments use TSDB/schema v13 unless compatibility requires otherwise
- Labels are intentionally low-cardinality
- High-cardinality context uses structured metadata/log fields
- OTLP attribute-to-label mapping is reviewed
- Retention is explicitly configured
- Object-store lifecycle exceeds Loki retention requirements
- Tenant identity is set only by trusted infrastructure
- Per-tenant ingestion/query limits exist
- Logs are structured and sensitive fields are redacted
- Trace IDs correlate logs with traces
- Query and ingestion capacity are monitored separately
- Recovery/object-storage durability is understood
Final Takeaway
Loki works best when you embrace its model instead of treating it like a traditional full-text search cluster.
Keep indexed labels small and bounded. Put high-cardinality context in structured metadata. Use modern TSDB/object-storage architecture, configure retention explicitly, enforce tenant identity outside Loki, and design LogQL queries around meaningful stream selectors.
The result is a logging system that can scale without turning every unique request ID into an index entry.
References
BUILDING AN AI PRODUCT?
Talk to Lofingo about custom autonomous AI agents, enterprise workflow automation, and scalable software systems.
EXPLORE AI SERVICES →