Apache Kafka is a distributed event streaming platform for writing, storing, reading, and processing continuous streams of events across multiple machines.
What Is an Event?
An event is a record describing something that happened:
OrderPlaced
PaymentCaptured
DeviceLocationUpdated
Applications called producers write events to Kafka.
Topics and Partitions
Events are organized into topics such as orders or payments.
Topics are split into partitions so data and traffic can be distributed across Kafka brokers. Events with the same key can be kept in the same partition, giving consumers ordered records within that partition.
Consumers
Consumers read topic records. Consumers sharing a group divide partitions between themselves, while a different consumer group can independently read the same event history.
That is useful when fulfillment, analytics, and search all need the same OrderPlaced event for different reasons.
Kafka Keeps Events After They Are Read
Unlike a traditional queue where an acknowledged message usually leaves the queue, Kafka retains records according to topic policy.
Consumers track their own positions using offsets, so events can be replayed while they remain retained.
Why Is Kafka Distributed?
Partitions and replicas are spread across brokers to increase throughput and tolerate failures. Modern Kafka also uses KRaft controllers to coordinate cluster metadata.
Where Kafka Fits
Kafka is commonly useful for:
- event-driven services
- change data capture
- analytics/event pipelines
- log/event aggregation
- stream processing
- integrations with many independent consumers
For one simple background job queue, a smaller queue system may be easier to operate.
Final Takeaway
Think of Kafka as a durable distributed event log. Producers append events to partitioned topics, consumers read them independently, offsets track progress, and retained records can be replayed later.
References
Discussion (0)