Apache Kafka Explained Simply: Topics, Partitions, Producers, Consumers, and Offsets

Apache Kafka Explained Simply: Topics, Partitions, Producers, Consumers, and Offsets

Apache Kafka is a distributed event streaming platform. Applications publish events to Kafka, Kafka stores them durably, and one or more consumers read those events independently.

Event

An event is one record, such as:

OrderPlaced
user_id=42
order_id=918

It can contain a key, value, timestamp, and optional headers.

Topic

A topic is the named stream where related events are written—for example orders or payments.

Unlike a traditional one-time queue, Kafka keeps records according to a retention policy even after consumers read them. Consumers can therefore replay older events while they still exist.

Partition

Topics are split into partitions across brokers. Partitions provide parallelism and scale.

Events with the same key are normally routed consistently so related records can stay together. Kafka guarantees ordering within one topic-partition, not one global order across every partition.

Producer

A producer publishes events to topics. It may choose a key so Kafka can select an appropriate partition.

Multiple producers can publish to the same topic.

Consumer and Consumer Group

Consumers read events. Consumers sharing the same group cooperate to process one logical subscription.

Topic with 4 partitions
Consumer group with 2 consumers
→ each consumer processes some partitions

Different consumer groups can independently read the same topic for different purposes such as analytics and fulfillment.

Offset

An offset identifies a record's position inside a partition. Consumer groups track their progress using offsets.

Because the data remains in Kafka according to retention, a consumer can move/replay from an earlier offset when the use case requires it.

Broker and Cluster

A Kafka broker is one server in the cluster. Topic partitions are distributed and replicated across brokers for capacity and fault tolerance.

Modern Apache Kafka uses KRaft for cluster metadata/controller coordination rather than ZooKeeper.

Why Kafka Is Useful

Kafka is a strong fit for:

• event-driven systems
• change-data capture
• application event pipelines
• analytics streams
• log/event aggregation
• stream processing

For a small background job queue with one consumer pool, a traditional queue may be simpler.

Final Takeaway

Learn Kafka through seven ideas: event → topic → partition → producer → consumer group → offset → broker. The key difference from a simple queue is that Kafka stores a durable event log that independent consumers can process and replay at their own pace.

References

• Apache Kafka 4.3 Getting Started
• Apache Kafka Introduction