Kafka Producers and Consumers: Acks, Batching, Consumer Groups, Offsets, and Rebalances
Kafka clients have two main jobs: producers append records to topics; consumers read partition logs and track their progress.
Producer: Choose Keys and Durability Deliberately
A producer can set a key so related records route consistently to a partition and preserve ordering within that partition.
For important events, use appropriate acknowledgements and idempotent-producer behavior rather than fire-and-forget publishing. Modern Kafka producer defaults enable idempotence when compatible settings are used.
Batch and Compress
Kafka producers are designed to batch records before sending them. Small amounts of extra batching latency can significantly improve network and broker efficiency.
Compression such as lz4 or zstd can reduce network/storage usage when the workload compresses well. Measure CPU vs bandwidth trade-offs with real payloads.
Consumer Groups Split Work
Consumers with the same group ID cooperate on one logical subscription. Partitions are assigned across group members, which creates horizontal processing parallelism.
Different consumer groups can independently read the same topic.
Offsets Are Processing Progress
Each consumer group records offsets for the partitions it processes. After restart or reassignment, consumers can resume from committed progress.
Commit strategy should match business processing. Committing before a side effect is safely completed can lose work; committing later can lead to replay, so handlers should be idempotent.
Rebalances Move Partitions Between Consumers
When group membership changes, partition assignments can move. Applications must finish, stop, or hand off work safely during reassignment.
Kafka's newer consumer rebalance protocol has been GA since Kafka 4.0 and uses incremental server-driven assignment; in Kafka 4.3 clients it remains opt-in with group.protocol=consumer.
Do not enable a new group protocol blindly if your client stack or custom assignment behavior has not been validated.
Shut Consumers Down Gracefully
On deployment or termination:
stop accepting new work
→ finish/abort current processing safely
→ commit appropriate progress
→ close consumer
This reduces unnecessary reprocessing and long rebalance delays.
Monitor Both Sides
Producer signals:
send error rate
request latency
record retry rate
batch/compression efficiency
Consumer signals:
consumer lag
processing latency
rebalance frequency
commit failures
poll/heartbeat health
Final Takeaway
Kafka producers and consumers are not just send() and poll(). Design partition keys, acknowledgements, batching, group membership, offset commits, idempotent processing, and rebalance behavior together so delivery remains predictable during load and failures.
References
BUILDING A DIGITAL PRODUCT?
Talk to Lofingo about modern web apps, high-performance software engineering, and production digital platforms.
EXPLORE SERVICES →