Apache Kafka Architecture in 2026: KRaft Controllers, Brokers, Partitions, Replication, and ISR
Modern Apache Kafka uses KRaft for cluster metadata and controller consensus. Kafka 4.3 supports KRaft only; ZooKeeper mode has been removed.
Brokers Handle Data Traffic
Brokers store topic partitions and serve producer/consumer requests. A Kafka cluster spreads partitions across brokers so storage and throughput can scale horizontally.
Clients bootstrap from one or more broker addresses, receive cluster/topic metadata, and then communicate with the brokers responsible for their partitions.
KRaft Controllers Manage Cluster Metadata
Controllers maintain metadata such as brokers, topics, partition assignments, and leadership in a replicated metadata log.
A controller quorum elects one active metadata leader while other controllers replicate that log. Production deployments should use a fault-tolerant quorum rather than one controller process.
Kafka 4.3 also supports dynamic KRaft quorum capabilities, so operators should follow current version-specific controller guidance rather than older ZooKeeper-era tutorials.
Partitions Have Leaders and Replicas
Each topic partition is replicated across brokers according to its replication factor.
Partition 0
leader: Broker A
followers: Broker B, Broker C
Producers write through the leader; followers replicate the partition log. If the leader fails, Kafka can elect an eligible replica according to the partition's replication state and controller decisions.
ISR and Durability
The in-sync replica (ISR) set represents replicas sufficiently caught up with the leader for normal replication guarantees.
A common durability design combines:
replication.factor = 3
min.insync.replicas = 2
producer acks = all
Exact values should follow your failure and availability requirements. Stronger durability can reject writes when too few replicas remain rather than accepting unsafe writes.
Kafka 4.x also introduced Eligible Leader Replicas (ELR), which expands safe leadership options in some degraded states while respecting Kafka's replication rules.
Rack Awareness Matters
Configure broker rack/zone identity so replicas of one partition can be spread across independent failure domains.
Three replicas on three brokers inside one physical failure domain are less useful than replicas distributed across zones/racks.
Monitor Both Metadata and Data Planes
Watch:
controller quorum leader / lag
partition leaders and ISR
under-replicated partitions
broker throughput and disk
request latency
consumer lag
A healthy controller quorum does not mean every data partition is healthy, and vice versa.
Final Takeaway
Modern Kafka architecture has two clear responsibilities: KRaft controllers coordinate cluster metadata; brokers store and serve partition data. Topics are partitioned, partitions are replicated, leaders handle traffic, and ISR/replication settings determine how the system behaves during failures.
References
BUILDING A SAAS PRODUCT?
Explore Lofingo’s end-to-end SaaS platforms, multi-tenant cloud architecture, and high-concurrency backend engineering.
EXPLORE SAAS SERVICES →