Kafka Topics Explained: Naming, Partitions, Retention, Compaction, and Ownership

Kafka Topics Explained: Naming, Partitions, Retention, Compaction, and Ownership

A Kafka topic is a named durable stream of records. Good topic design defines more than a name: it defines ownership, partitions, replication, retention, cleanup behavior, and the event contract consumers depend on.

Design Topics Around Stable Data Meaning

Prefer names tied to a clear domain/event family:

orders.events
payments.events
inventory.changes

Avoid one giant events topic containing unrelated schemas and ownership. Consumers should know which team owns the contract and what records mean.

Create Important Topics Explicitly

Kafka supports automatic topic creation, but current Kafka Streams guidance recommends not relying on broker auto-creation for user topics because defaults such as replication factor or partition count may not match production requirements.

Provision important topics through infrastructure/configuration code with reviewed settings.

Partitions Define Parallelism and Ordering

A topic is divided into partitions. Partition count affects write/read distribution and maximum traditional consumer-group parallelism.

Choose a partition key based on the ordering scope you need, and remember that increasing partition count later can change key placement for the default partitioner.

Replication Defines Fault Tolerance

Replication factor controls how many brokers hold each partition's data. A common production design uses multiple replicas across racks/availability zones, together with suitable min.insync.replicas and producer acknowledgement settings.

Retention vs Compaction

Kafka provides two main cleanup models:

Delete retention

Old log segments are deleted according to configured time/size limits. Use this for event history that only needs a bounded replay window.

Log compaction

Kafka retains the latest value for each key while older superseded records can be removed.

This fits changelog/state topics where consumers need the latest known state per key rather than every historical update forever.

A topic can use both delete and compact when both behaviors are useful.

Schemas Are Part of the Topic Contract

Define serialization and compatibility rules for important topics. Prefer additive evolution and validate changes before deploying producers that older consumers cannot read.

Topic ACLs should also restrict who can publish, consume, or administer sensitive streams.

Final Takeaway

A Kafka topic is a durable shared data contract. Create important topics explicitly, assign clear ownership, choose partitions and replication deliberately, set retention/compaction from replay needs, and evolve schemas without surprising independent consumers.

References

• Apache Kafka 4.3 Basic Operations
• Apache Kafka Broker Configs
• Kafka Streams Topic Management