Prometheus in Production: PromQL, Metrics, Histograms, Cardinality, and Scaling

Prometheus in Production: PromQL, Metrics, Histograms, Cardinality, and Scaling

Prometheus is a time-series monitoring system built around a simple model: applications expose numeric metrics, Prometheus scrapes them, and operators use PromQL to query behavior over time.

The difficult part is not installing Prometheus. It is designing metrics that remain useful without creating millions of expensive time series.

The Data Model

Prometheus stores time series identified by a metric name plus labels:

http_requests_total{
  method="POST",
  route="/orders/{id}",
  status="201"
}

Every unique label combination is a separate series.

That makes labels powerful and dangerous.

Choose Metrics by Meaning

Common metric types include:

Counter

A value that only increases except on process reset:

http_requests_total
payments_failed_total

Use rate() or increase() to analyze counters over time.

Gauge

A value that can rise and fall:

queue_depth
active_connections
memory_bytes

Histogram

Represents a distribution such as request duration or response size.

Prometheus now documents native histograms as the preferred histogram form when your client/backend support them, while classic histograms remain widely used.

Histograms allow server-side percentile-style analysis and aggregation across instances.

Name Metrics Predictably

Prometheus recommends:

• a meaningful subsystem/application prefix where appropriate
• one quantity per metric
• base units such as seconds and bytes
• suffixes that communicate units/type, such as _seconds and _total

Good:

http_request_duration_seconds
orders_created_total
process_memory_bytes

Ambiguous metric names make every PromQL query harder to review during an incident.

Cardinality Is the Main Metric Design Trap

Never use unbounded labels such as:

user_id
email
request_id
session_id
order_id
full URL with IDs

A metric with 10 routes × 5 status codes × 3 regions creates 150 combinations.

Add one million users as a label and the design explodes.

Prometheus's own instrumentation guidance strongly recommends keeping most metrics low-cardinality and moving high-cardinality analysis to logs/traces or other systems.

Use route templates:

/orders/{id}

not raw paths:

/orders/928183

PromQL Turns Counters Into Useful Rates

A counter by itself gives lifetime totals. PromQL lets you derive operational meaning.

Request rate:

sum(rate(http_requests_total[5m]))

5xx ratio:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

The important part is understanding the metric semantics. A syntactically valid query can still produce misleading monitoring if you aggregate labels incorrectly.

Use Histograms for Latency SLOs

For latency-sensitive services, you need distributions—not one average.

A classic histogram exposes bucket counts. Native histograms encode distributions more directly.

Use them to answer questions such as:

What percentage of requests finished under the SLO threshold?
How is p95/p99 latency changing?
Which route is producing the long tail?

Choose buckets/schema based on real latency ranges, not arbitrary copy-paste defaults.

Recording Rules Precompute Expensive Queries

If the same complex PromQL expression powers many dashboards and alerts, a recording rule can periodically save the result as a new time series.

This reduces repeated query cost and creates stable higher-level metrics for SLO dashboards.

Alert Rules Should Describe Symptoms

Prometheus alert rules evaluate PromQL conditions. Route notifications through Alertmanager or an equivalent alerting layer.

Good alerts should include:

• service/owner labels
• severity
• clear annotation
• dashboard/runbook link
• enough duration to avoid transient noise

Prefer user-visible symptoms and error-budget burn over dozens of arbitrary infrastructure thresholds.

Service Discovery Keeps Scrape Targets Dynamic

Prometheus can discover targets from environments such as Kubernetes, cloud providers, and service registries.

That is essential when instances are ephemeral:

new Pod appears → discovered → scraped
old Pod disappears → removed

Avoid static lists of container IPs in dynamic infrastructure.

Scrape Interval Is a Cost/Resolution Trade-Off

A shorter interval gives finer resolution but creates more samples, storage, and network/CPU work.

Do not scrape every metric every second simply because Prometheus allows it.

Choose intervals based on how quickly the signal changes and how rapidly operators need to detect it.

Prometheus Is Not a High-Cardinality Event Database

Do not use metrics to store:

individual request IDs
user activity history
audit events
raw stack traces
business records

Metrics summarize behavior. Logs and traces preserve event-level context.

Trying to turn Prometheus into a general-purpose event store creates cardinality and retention problems.

High Availability Means Duplicate Prometheus Servers

A common HA pattern runs multiple Prometheus servers scraping the same targets independently.

This avoids a single Prometheus process becoming the only source of monitoring data.

Long-term/global query systems can deduplicate replicas depending on the chosen architecture.

Remote Write Extends Storage and Aggregation

Prometheus can send samples to remote systems through remote_write.

This is useful for:

• long retention
• centralized metrics across clusters
• managed backends
• horizontally scalable systems such as Mimir/Thanos-style architectures

Remote write has its own queues and backpressure behavior. Prometheus documentation notes that sustained remote endpoint outages can eventually exceed WAL retention, so monitor remote-write lag and failures rather than assuming the pipeline is lossless forever.

Local Retention Should Be Intentional

A standalone Prometheus server uses local TSDB storage. Retention is typically bounded by time and/or size.

For very long retention or global multi-cluster querying, use architecture designed for that workload instead of turning one Prometheus node into an enormous stateful server.

Monitor Prometheus Itself

Watch:

• scrape failures
• sample ingestion rate
• active series
• rule evaluation failures/duration
• TSDB disk usage
• WAL/storage errors
• remote-write backlog/failures
• query latency

Your monitoring system is production infrastructure too.

Production Checklist

• Metric names use clear units and semantics
• Counters/gauges/histograms are chosen correctly
• Native histograms are evaluated where supported
• High-cardinality labels are prohibited
• Route labels use templates, not raw IDs
• PromQL expressions are reviewed for correct aggregation
• Repeated expensive queries use recording rules
• Alerts are actionable and SLO-oriented
• Dynamic targets use service discovery
• Scrape intervals match operational needs
• Prometheus HA is planned for critical monitoring
• Remote-write lag/failure is monitored
• Retention and disk capacity are explicit

Final Takeaway

Prometheus is extremely effective when metrics remain small, numeric, and intentionally dimensional.

Model the behavior you want to measure, keep labels bounded, use histograms for distributions, precompute expensive queries, and connect alerts to user-facing reliability. Scale beyond one Prometheus server through HA and remote/global storage patterns only when retention or fleet size requires it.

Most Prometheus problems are not caused by PromQL. They begin when a metric label quietly turns every user, request, or object into a new time series.

References

• Prometheus Documentation
• Metric and Label Naming
• Instrumentation Best Practices
• Histograms and Summaries
• Recording Rules
• Remote Write Tuning