Apache Cassandra in Production: Data Modeling, Consistency, Compaction, and Hot Partitions

Apache Cassandra in Production: Data Modeling, Consistency, Compaction, and Hot Partitions

Apache Cassandra is a distributed wide-column NoSQL database built for workloads that need horizontal scale, high availability, multi-node replication, and predictable key-oriented access.

Its biggest difference from a relational database is architectural: you model tables around the queries you already know you need. Cassandra works extremely well when partitions are well distributed and access patterns are explicit. It becomes awkward when an application expects arbitrary joins, ad-hoc filtering, or one normalized schema to answer every future query.

Cassandra's architecture in one minute

Cassandra has no single permanent primary for the whole cluster. Data is partitioned across nodes and replicated according to the keyspace's replication strategy.

partition key
     |
     v
hash / token
     |
     +--> replica node A
     +--> replica node B
     +--> replica node C

A client can contact a Cassandra node as a coordinator. The coordinator finds the replicas that own the requested partition and contacts enough of them to satisfy the chosen consistency level.

This architecture is one reason Cassandra can continue serving many workloads through individual node failures.

The partition key decides data placement

Every table has a primary key whose first part is the partition key.

CREATE TABLE orders_by_customer (
    customer_id uuid,
    created_at timestamp,
    order_id uuid,
    total decimal,
    PRIMARY KEY ((customer_id), created_at, order_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

Here:

• customer_id is the partition key
• created_at and order_id are clustering columns

Rows for one customer are colocated on the same replica set, making this query natural:

SELECT *
FROM orders_by_customer
WHERE customer_id = ?
LIMIT 50;

Model tables from queries, not entities

A relational model might keep one canonical orders table and depend on joins and indexes for many query shapes.

Cassandra often uses denormalized tables dedicated to important access patterns:

orders_by_customer
orders_by_status_and_day
order_by_id

That duplication is deliberate. It avoids expensive distributed joins and broad scans.

A useful rule is: each Cassandra table should have a clear set of partition-key-addressable queries it exists to serve.

Keep partitions bounded

A partition groups data that should be read together, but it should not grow forever.

A risky model is:

partition key = tenant_id

when a large tenant can accumulate an enormous history under one key. That can create oversized partitions, hot replica sets, expensive compaction, and unpredictable reads.

Time bucketing is a common fix:

partition key = (tenant_id, day)

or:

partition key = (device_id, month)

Choose the bucket from real data volume and query windows. The goal is neither “tiny partitions” nor “huge partitions,” but predictable bounded partitions.

Avoid hot partition keys

Even a reasonably sized partition can become a traffic hotspot if one key receives disproportionate requests.

Examples include:

• one celebrity account
• one live event
• a low-cardinality status such as OPEN
• one global counter

Adding nodes does not automatically split one logical Cassandra partition across more owners. The model may need bucketing, striped counters, tenant subdivision, or another distribution dimension.

Horizontal scale requires horizontally distributable work.

Clustering columns organize data inside a partition

Clustering columns determine row ordering inside each partition.

For the orders table:

customer_id -> partition
created_at  -> clustering order
order_id    -> tie-breaker / uniqueness

That enables bounded range queries such as “recent orders for this customer between two timestamps” without scanning unrelated partitions.

Good clustering-key design is how Cassandra turns a known access pattern into efficient sequential reads.

Consistency is tunable per operation

Cassandra is eventually consistent across replicas by default, but clients choose consistency levels for reads and writes.

Common levels include:

• ONE
• QUORUM
• LOCAL_QUORUM
• ALL

Requiring more replicas generally improves agreement/freshness at the cost of more coordination and potentially lower availability during failures.

For multi-datacenter deployments, local consistency levels are often used so normal requests do not wait for a distant datacenter.

Choose consistency from the business invariant, not one cluster-wide rule copied from another system.

Lightweight Transactions are for compare-and-set

Cassandra Lightweight Transactions (LWT) provide linearizable compare-and-set behavior for operations such as:

INSERT ... IF NOT EXISTS;
UPDATE ... IF version = 7;

LWT uses additional consensus coordination and is materially more expensive than a normal Cassandra write.

Use it for invariants that truly need conditional linearizable behavior. Do not make every update an LWT merely “for safety.”

The storage engine is append-oriented

A simplified write path is:

client write
   |
commit log
   |
memtable
   |
flush
   v
immutable SSTable

Updates do not normally rewrite old SSTables in place. New versions are written, and background compaction later merges SSTables.

This design supports high write throughput, but it makes compaction and tombstone behavior first-class operational concerns.

Compaction is essential

As memtables flush, Cassandra accumulates immutable SSTables. Reads can need several SSTables to reconstruct the newest row state.

Compaction merges them to:

• reduce duplicate/obsolete versions
• reclaim storage
• remove safe tombstones
• reduce read amplification

Different compaction strategies fit different workloads. The best choice depends on read/write shape, TTL use, time-series behavior, and retention—not one universal default.

Tombstones represent deletion

Because SSTables are immutable, deletion writes a timestamped marker called a tombstone instead of instantly removing older bytes.

TTL expiration also eventually produces deletion state that compaction must reclaim.

Excessive tombstones can hurt reads, especially when queries walk through large ranges of expired/deleted data.

Common causes include:

• delete-heavy workloads
• very short TTLs over huge datasets
• weak partition/query models
• compaction choices that do not match the lifecycle

Model retention and queries together.

TTL is useful, but storage cleanup is not immediate

Cassandra supports TTL for naturally temporary data such as sessions, telemetry, idempotency state, and bounded event retention.

INSERT INTO session_by_id (session_id, user_id, value)
VALUES (?, ?, ?)
USING TTL 1800;

After expiry, data participates in Cassandra's tombstone and compaction lifecycle. Heavy TTL workloads therefore need compaction and disk-space planning, not just an expiration number.

Replication factor affects durability, availability, and cost

A keyspace's replication strategy determines how many copies of each partition exist and where they live.

Replication factor affects:

• tolerated replica failures
• storage multiplier
• network traffic
• repair work
• available consistency levels

For topology-aware production deployments, use a strategy designed for the real datacenter layout rather than a single-node development configuration.

More replicas increase resilience but are not free.

Repair keeps replicas converged

A replica can miss changes while unavailable. Cassandra repair compares replicas and synchronizes inconsistent ranges.

Production clusters need a deliberate repair process aligned with:

• maximum node outage duration
• data volume
• tombstone grace settings
• repair frequency
• cluster topology

Replication without a repair strategy is incomplete operations design.

Modern Cassandra includes Storage-Attached Indexing

Storage-Attached Indexing (SAI) provides modern secondary-index capabilities and is also the foundation for Cassandra's vector-search support.

SAI can expand practical query options, but it does not erase the importance of good primary-key modeling. Measure index selectivity, result cardinality, write overhead, and hot-key behavior under production-like data.

Indexes should complement the data model, not rescue a table with no clear access pattern.

Cassandra 5.0 added vector search

Current Apache Cassandra documentation includes a vector data type and SAI-based approximate nearest-neighbor search, introduced in Cassandra 5.0.

A table can conceptually store:

tenant_id
record_id
embedding: fixed-dimensional float vector
content

and index the embedding for similarity search.

This can be useful when an existing Cassandra workload also needs semantic retrieval. It does not mean Cassandra is automatically the best vector database for every AI workload—compare filtering, recall, update rate, index size, latency, and operations against specialist alternatives.

Do not use ALLOW FILTERING as a schema strategy

ALLOW FILTERING can make some otherwise unsupported CQL queries execute by evaluating more data.

That may be acceptable for bounded administrative work. It is a poor foundation for a high-traffic endpoint over a large dataset.

If an important query cannot be served cleanly, redesign the table/index or use a database better suited to that query shape.

When Cassandra is a strong fit

Cassandra is attractive when:

• write volume is high and distributable
• queries are predictable and key-oriented
• multi-node/multi-datacenter availability matters
• denormalization is acceptable
• partitions can be kept well distributed and bounded

A relational database is usually simpler when the application needs frequent arbitrary joins, evolving ad-hoc queries, rich relational constraints, or complex multi-row transactions.

CQL looks SQL-like; the architecture is not relational SQL underneath.

What to monitor

Track at least:

• read/write p95/p99 latency
• coordinator timeouts/unavailable errors
• dropped messages
• pending compactions
• SSTable counts
• tombstone-heavy reads
• disk usage per node
• partition-size distribution
• repair status
• node health
• JVM/GC pressure
• hot partitions

Cluster-wide averages can hide one overloaded partition or replica set.

Production checklist

Before shipping a Cassandra table, verify:

1. Every important query has a deliberate table/index path.
2. The partition key distributes traffic, not only rows.
3. Partition growth is bounded.
4. Clustering keys match range/order requirements.
5. Consistency levels match correctness needs.
6. LWT is reserved for operations that require it.
7. TTL and tombstone behavior are understood.
8. Compaction matches the table lifecycle.
9. Replication and repair procedures are defined.
10. SAI/vector workloads have been measured with realistic data.

Cassandra works best when you accept its core contract: query-first modeling, bounded distributed partitions, and explicit consistency/maintenance choices.

Official references

• Apache Cassandra architecture overview
• Cassandra data definition and primary keys
• Cassandra compaction
• Cassandra guarantees
• Cassandra Vector Search