Setting Up a Multi-Node KRaft-based Kafka Cluster – A Practical Guide

Learn how to setup a multi-node Kafka cluster, with 3 dedicated controller nodes, and 3 dedicated broker nodes for enhanced fault tolerance using Docker. The cluster will be configured using Apache Kafka version 3.7.1, which uses the new Kafka Raft (KRaft) mode, without the traditional Zookeeper that is soon going to be phased-out starting with Kafka version 4.0.

If you want to start with something simpler, you might want to check out my other blog that describes setting up a single-node Kafka cluster here – Setting Up a Single-Node Kafka Cluster using KRaft Mode – No More Zookeeper Dependency

Introduction to Kafka Raft (KRaft)

Apache Kafka is a powerful distributed event streaming platform that can handle high throughput and scalability. One of the newer features in the Kafka ecosystem is KRaft (Kafka Raft Consensus Protocol), which simplifies metadata and cluster management without relying on Zookeeper.

In this blog post, we will explore how to create a 6-node KRaft-based Kafka cluster using Docker.

Overview of the Cluster

We will set up a 6-node Kafka cluster, that will have 3 dedicated controller nodes and 3 dedicated broker nodes. Controller nodes manage the metadata of the cluster, its configuration, state, etc. whereas broker nodes store and serve actual data/messages to clients (producers and consumers).

The cluster we will setup, at its simplest will look like this –

Things to Note

• Simple setup with minimum configuration – This blog focuses on setting up a working Kafka cluster with KRaft mode and the minimum configuration it requires. The goal is to help readers get a functional cluster up and running quickly, without diving into advanced configurations or handling all potential corner cases.
• All 6 nodes on the same machine – In this setup, all 6 nodes are deployed on the same host machine using Docker for simplicity and ease of demonstration. However, for production environments, it is strongly recommended to deploy each node on a separate machine to ensure fault tolerance and performance.
• No SSL, Authentication, Authorization – This setup does not include SSL, authentication, or authorization configurations to keep the focus on getting a functional Kafka cluster running in KRaft mode.
• No persistent storage – This setup does not include persistent storage, meaning data will not survive a container restart.
• Host network – This setup uses the host network for simplicity, which allows all containers to share the same network stack as the host machine. While this simplifies configuration, it is generally recommended to use a dedicated network setup for better isolation and security in production environments.
• Listener ports – In this setup, the 3 controller nodes use ports 29091, 29092 and 29093 respectively. Similarly the 3 broker nodes use ports 29094, 29095 and 29096.

Prerequisites for Creating the Cluster

1. Docker – Before diving into the setup, ensure you have Docker installed on your machine. Familiarity with Docker commands will be advantageous.
2. Apache Kafka Docker Image – This setup uses Docker for deploying all controller and broker nodes of the cluster. It uses Apache Kafka docker image with version 3.7.1 which can be pulled from docker hub using following command –
   • docker pull apache/kafka:3.7.1

Step-by-Step Guide to Deploying the Kafka Cluster

Step 1 – Generate a Cluster ID

The Cluster ID serves as a globally unique identifier for a Kafka cluster, distinguishing it from other clusters. All controllers and brokers must share the same Cluster ID to function as part of the same cluster.

Generate a cluster ID using the kafka-storage.sh script with random-uuid argument as follows –

docker run --rm \
apache/kafka:3.7.1 \
/opt/kafka/bin/kafka-storage.sh random-uuid

This will return a random UUID such as “zn1DViu4S1e2pbQXRYrOqA”. We will use this UUID as cluster ID in the rest of the steps.

Step 2 – Deploy Controller 1

Deploy and run 1st controller node using following docker command –

docker run --name controller_1 \
--network host \
-e KAFKA_NODE_ID=1 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=controller \
-e KAFKA_LISTENERS=CONTROLLER://localhost:29091 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.1

Explanation of the env variables –

• KAFKA_NODE_ID – Unique ID for each node. For the current setup, it will be 1, 2 and 3 for controllers and 4, 5 and 6 for broker nodes. See official documentation here.
• CLUSTER_ID – This should be consistent across all the nodes (controllers and brokers) in the cluster as described in Step 1 above.
• KAFKA_PROCESS_ROLES – This should be equal to “controller” for all controller nodes, and “broker” for all broker nodes. It is also possible to have a node that acts as both controller and broker in which case its value will be “broker,controller”. Such a dual role is suitable for development/testing but not recommended for production. For our current setup, we will have each node with a dedicated role. See official documentation here.
• KAFKA_LISTENERS – The host and port on which the node will be listening for requests. The word “CONTROLLER” here is the name of the listener and is defined in the next config parameter KAFKA_CONTROLLER_LISTENER_NAMES. See official documentation here.
• KAFKA_CONTROLLER_LISTENER_NAMES – Name of the listener used for all controller traffic. This could be any name of your choice provided you use the same name in the dependent config parameters (such as KAFKA_LISTENERS). See official documentation here.
• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP – Security protocol to use for each listener. For this setup, we have 1 listener named CONTROLLER, and we will be using PLAINTEXT (i.e. non-encrypted traffic) for this listener. See official documentation here.
• KAFKA_CONTROLLER_QUORUM_VOTERS – This property lists the nodes responsible for storing and managing the cluster’s metadata using the Raft consensus protocol. The nodes are listed using their node ID and their address in host/port format. Here we provide details of all the 3 controller nodes since they constitute the quorum. This configuration should be same for all nodes (controllers and brokers). See official documentation here.

Once the 1st controller starts, it tries to reach other controller nodes 2 and 3. Since they are not yet deployed, you will notice following error messages rapidly printed in the log. These error messages should stop once the remaining controller nodes are also deployed and running and are not a cause for concern –

Connection to node 2 (localhost/127.0.0.1:29092) could not be established. Node may not be available (org.apache.kafka.clients.NetworkClient)

Connection to node 3 (localhost/127.0.0.1:29093) could not be established. Node may not be available (org.apache.kafka.clients.NetworkClient)

Step 3 – Deploy Controller 2 and 3

Once 1st controller is up, deploy 2nd and 3rd controller on the similar lines using following docker commands –

Controller 2 –

docker run --name controller_2 \
--network host \
-e KAFKA_NODE_ID=2 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=controller \
-e KAFKA_LISTENERS=CONTROLLER://localhost:29092 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.1

Controller 3 –

docker run --name controller_3 \
--network host \
-e KAFKA_NODE_ID=3 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=controller \
-e KAFKA_LISTENERS=CONTROLLER://localhost:29093 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.1

Step 4 – Deploy Broker 1

Deploy and run 1st broker node using following docker command –

docker run --name broker_1 \
--network host \
-e KAFKA_NODE_ID=4 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=broker \
-e KAFKA_LISTENERS=BROKER://localhost:29094 \
-e KAFKA_ADVERTISED_LISTENERS=BROKER://localhost:29094 \
-e KAFKA_INTER_BROKER_LISTENER_NAME=BROKER \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,BROKER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.1

Explanation of the env variables –

• KAFKA_PROCESS_ROLES – This should be “broker” for all broker nodes.
• KAFKA_LISTENERS – Here the word “BROKER” is the name of the listener that will be used for listening for all external traffic reaching to the broker.
• KAFKA_ADVERTISED_LISTENERS – For the current setup, this is same as KAFKA_LISTENERS, however it can be different depending on the network setup. See official documentation here.
• KAFKA_INTER_BROKER_LISTENER_NAME – Name of the listener that will be used for inter-broker communication such as during data replication. For the current setup, we are using same listener named “BROKER” for external as well as inter-broker traffic. See official documentation here.
• KAFKA_CONTROLLER_LISTENER_NAMES – For broker nodes, this specifies the listener on broker nodes used for communication with the controllers.
• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP – Here we define the security protocol for both the listeners i.e. BROKER – responsible for external traffic from clients and inter-broker traffic, and CONTROLLER – responsible for broker to controller traffic. For the current setup, both listeners use PLAINTEXT i.e. non-encrypted traffic.
• KAFKA_CONTROLLER_QUORUM_VOTERS – This property lists all the controller nodes. This configuration is same for all controller as well as broker nodes.

Step 5 – Deploy Brokers 2 and 3

Once broker 1 is deployed, deploy remaining broker nodes 2 and 3 on similar lines using following commands –

Broker 2 –

docker run --name broker_2 \
--network host \
-e KAFKA_NODE_ID=5 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=broker \
-e KAFKA_LISTENERS=BROKER://localhost:29095 \
-e KAFKA_ADVERTISED_LISTENERS=BROKER://localhost:29095 \
-e KAFKA_INTER_BROKER_LISTENER_NAME=BROKER \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,BROKER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.1

Broker 3 –

docker run --name broker_3 \
--network host \
-e KAFKA_NODE_ID=6 \
-e CLUSTER_ID=zn1DViu4S1e2pbQXRYrOqA \
-e KAFKA_PROCESS_ROLES=broker \
-e KAFKA_LISTENERS=BROKER://localhost:29096 \
-e KAFKA_ADVERTISED_LISTENERS=BROKER://localhost:29096 \
-e KAFKA_INTER_BROKER_LISTENER_NAME=BROKER \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,BROKER:PLAINTEXT \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:29091,2@localhost:29092,3@localhost:29093 \
apache/kafka:3.7.

The above steps complete the cluster setup part. Next you may want to verify the status of the cluster by following the next step.

Step 6 – Verify Cluster Status

Verify the status of the cluster by running kafka-metadata-quorum.sh script with argument describe --status as follows –

docker run --rm \
--network host \
apache/kafka:3.7.1 \
/opt/kafka/bin/kafka-metadata-quorum.sh \
--bootstrap-server localhost:29094 \
describe --status

Note that for the --bootstrap-server argument, we always provide connection details of the BROKER node.

Running the above command returns current status of the cluster that includes the Cluster ID, the ID of the current leader controller, the IDs of all the controller and broker nodes, etc. as follows –

ClusterId:               zn1DViu4S1e2pbQXRYrOqA
LeaderId:                1
LeaderEpoch:             5
HighWatermark:           854
MaxFollowerLag:          0
MaxFollowerLagTimeMs:    0
CurrentVoters:           [1,2,3]
CurrentObservers:        [4,5,6]

Here, “CurrentVoters” refers to the quorum voters, i.e. controller nodes, and “CurrentObservers” refers to the broker nodes.

Next you might also want to run a console-consumer and a console-producer to test the cluster by following the below optional steps.

Optional Step 7 – Run a Console-Producer to Send Messages

Run a console-producer application using following docker command. Note that the same docker image used for creating the cluster can be used for running clients such as console-producer and console-consumer. No additional setup is required to run console clients.

docker run --rm -it \
--network host \
apache/kafka:3.7.1 \
/opt/kafka/bin/kafka-console-producer.sh --topic MY_FIRST_TOPIC \
--bootstrap-server localhost:29094

Once you run the above command, it returns a prompt where you can enter messages. Note the -it flag in the docker run command which means interactive mode. This is important especially for console producer since it prompts you and lets you enter new messages interactively.

Note that by default Kafka automatically creates topics when they are referenced by a producer or a consumer. This is governed by config parameter auto.create.topics.enable which by default is set to true.

If you prefer a programmatic Kafka producer using Java instead of a console producer, you can find a step-by-step guide on my other blog here – How to Build Your First Kafka Producer: A Step-by-Step Tutorial.

Optional Step 8 – Run a Console-Consumer to Receive Messages

In another terminal, start a console-consumer client using below docker command –

docker run --rm -it \
--network host \
apache/kafka:3.7.1 \
/opt/kafka/bin/kafka-console-consumer.sh --topic MY_FIRST_TOPIC \
--bootstrap-server localhost:29094

If everything is okay, you should start seeing messages sent by the console-producer application.

If you followed the above steps, and there are no errors in the logs, and if the console-consumer is able to receive messages sent by the console-producer, congratulations – you have deployed a functional Kafka cluster running in KRaft mode with the minimum required configuration, ready to handle basic workloads.

---

Thank You!

Thank you for Reading! I hope this guide helped you set up your Kafka cluster with ease. If you have any questions, feedback, or suggestions, feel free to share your thoughts in the comments. I’d love to hear what you think!