Kafka for Microservices: Topics, Consumer Groups, Idempotency, Outbox, and Event Contracts

Kafka for Microservices: Topics, Consumer Groups, Idempotency, Outbox, and Event Contracts

Kafka is useful in microservices when durable events need to outlive one request and be consumed independently by several services.

Publish Domain Events, Not Database Noise

Prefer events such as:

OrderPlaced
PaymentCaptured
InventoryReserved

with stable IDs, timestamps, schema/version rules, and the minimum data consumers need.

Do not expose every internal table mutation as a public service contract unless CDC is intentionally the integration model.

Choose Partition Keys From Ordering Requirements

Kafka preserves record order within a partition. If all events for one order must be processed in order, order_id is a natural key.

The trade-off is distribution: one extremely hot key stays on one partition. Partition design must balance ordering with throughput.

Consumer Groups Represent Independent Subscribers

All instances in one consumer group cooperate on one logical subscription. A different group receives its own view of the same retained events.

That makes it natural for several microservices to consume the same event stream without the producer sending separate copies manually.

Consumers Still Need Idempotency

Offsets and Kafka transactions can provide strong guarantees inside Kafka workflows, but your consumer may also write PostgreSQL, call a payment API, or send email.

Those external side effects still need idempotency or another reliable processing design. Do not label an entire cross-system workflow “exactly once” just because one Kafka client uses transactions.

Fix the Database + Event Dual-Write Problem

This is unsafe:

1. commit order to DB
2. publish OrderPlaced

A crash between the steps can lose the event.

Use the transactional outbox pattern or an appropriate CDC mechanism: commit business state and an outgoing-event record in one local database transaction, then publish asynchronously.

Evolve Event Schemas Carefully

Producers and consumers deploy independently. Prefer additive compatible changes, explicit schema ownership, and automated compatibility validation where useful.

Keep old consumers in mind before renaming/removing required fields.

Monitor Consumer Lag

Important Kafka signals include:

producer errors/latency
consumer lag
rebalance frequency
partition skew
under-replicated partitions
processing failures

Lag measures how far processing is behind the retained log; it should be tied to business freshness requirements, not one arbitrary threshold.

When Kafka Is Too Much

If one service simply needs to enqueue a small background job for one worker pool, a traditional queue can be operationally simpler. Kafka earns its complexity when durable replay, high-throughput streams, several independent consumers, or stream processing are actual requirements.

Final Takeaway

Kafka works well between microservices when you design domain events, partition keys, consumer groups, idempotent processing, reliable DB-to-event publication, and compatible schemas as one system. The broker moves records; application correctness still belongs to your services.

References

• Apache Kafka Introduction
• Apache Kafka Producer API
• AWS Transactional Outbox Pattern