Apache Kafka in Production: KRaft, Replication, Storage, Security, and Monitoring

Apache Kafka in Production: KRaft, Replication, Storage, Security, and Monitoring

Running Kafka in production is mostly about durability, disk/network capacity, controller health, replication, security, and operational visibility.

Run a Fault-Tolerant KRaft Control Plane

Kafka 4.3 is KRaft-only. Use a controller quorum that can tolerate controller failure and monitor metadata-log replication/leadership.

Keep controller responsibilities isolated and sized appropriately for larger clusters; losing broker capacity and losing metadata quorum are different failure classes.

Set Replication and Write Durability Deliberately

For important topics, a common baseline is multiple replicas across failure domains with producer acks=all and a suitable min.insync.replicas.

replication factor 3
+ min ISR 2
+ acks=all

means Kafka can reject writes rather than acknowledge data when too few safe replicas remain.

Kafka 4.x also supports Eligible Leader Replicas (ELR), enabled by default on new clusters since 4.1, to improve safe leadership availability in some degraded conditions.

Spread Replicas Across Racks or Zones

Configure broker.rack so replicas of the same partition are placed across independent racks/availability zones where possible.

Replication on three brokers in one shared failure domain does not protect against that domain failing.

Storage and Network Drive Broker Capacity

Kafka is a high-throughput log system. Monitor disk throughput/latency, page-cache effectiveness, network bandwidth, partition distribution, and storage headroom.

Avoid sharing Kafka data disks with unrelated heavy application logging or I/O workloads.

Tiered Storage Is Useful—but Know the Limits

Kafka 4.3 supports tiered storage for moving completed log segments to remote storage while keeping a smaller local hot set.

It can reduce broker-local disk needs for long retention, but current documentation still lists limitations including no support for compacted topics. Validate your workload before adopting it broadly.

Secure Every Connection Path

Kafka supports TLS plus SASL mechanisms and ACL-based authorization.

Production clusters should intentionally configure:

• encrypted client/broker traffic where required
• authenticated clients and brokers
• topic/group ACLs with least privilege
• restricted admin/controller network access
• secured JMX/management endpoints

An internal network is not a complete authorization strategy.

Monitor the Data and Metadata Planes

Track at minimum:

under-replicated partitions / ISR changes
broker bytes in/out and request latency
controller quorum leadership/lag
disk usage and I/O
producer errors/retries
consumer lag
rebalances
tiered-storage copy/fetch errors when enabled

Alert from business freshness and durability risk, not every low-level metric spike.

Plan Rebalancing and Growth

Adding brokers does not automatically move all existing data onto them. Partition reassignment and leadership balance need deliberate operational handling.

Likewise, changing partition counts can alter key distribution. Capacity planning should happen before disk or throughput becomes an emergency.

Final Takeaway

Production Kafka needs healthy KRaft controllers, replicated partitions across failure domains, enough disk/network headroom, secure client access, monitored lag/ISR, and planned capacity changes. Kafka is durable because those controls are operated correctly—not because the cluster has three broker processes.

References

• Apache Kafka 4.3 Operations
• Kafka Hardware and OS
• Kafka Security Overview
• Kafka Tiered Storage
• Kafka Monitoring