How to Integrate Custom Zookeeper with Kafka Cluster

The zookeeper is a popular configuration management and coordination centralized service supported by Apache. It manages the metadata based on a tree-like, hierarchical structure which make it respond quickly and integrally even in a huge distributed system like Kafka. It is required for Kafka by default, but most enterprises will have their own Zookeeper system. So, in this article, we will talk about how to build your custom Zookeeper system and how to integrate it with the Kafka cluster.

• Prerequisite

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

At first, it assumes the setup includes several servers – no less than 3 nodes. But for personal practice, I suggest you use Docker to build the Zookeeper cluster. You can use a single server to build a Zookeeper cluster. In this article, I will use Docker by default. Following is the environment requirement:

1. Server: a PC or Server with 8GB will be ok.
2. OS: Ubuntu 14.04 LTS + (Others that are compatible with Docker are ok)
3. Docker
4. An existing Docker Container with JDK 7.6+ installed.

Tips: if we want to use the Zookeeper on Docker, please make sure of the port mapping to the real server from the Docker Container.  A Zookeeper needs three ports to be set. One is for the external application to access to Zookeeper service. One is for the cluster internal communication. The third one is for the Zookeeper leadership service. So, the only thing we need to do is to publish the public Zookeeper service port by Docker Port Mapping (-p command).

• Build Zookeeper cluster

1. Download the Zookeeper installation compressed package from the Apache Official Site.
2. Use the docker cp command to copy the Zookeeper installation package into the Docker container.
3. Decompress the Zookeeper package into a folder in the Docker container.
4. Open the ‘Conf’ folder under the decompressed Zookeeper folder.
5. Rename or copy the zoo_sample.cfg file to zoo.cfg.
6. Open and edit the zoo.cfg file                              
7. Add myid file with the server identifier number under the data folder you set in the configuration file.
8. Tips: Each Zookeeper cluster node should have its unique identifier number which should be different from the others and mapped to one of the cluster node endpoints (server.identifier=server id: port1: port 2). For example, if you have three Zookeeper cluster nodes, when you configure the first one, you give it an identifier ‘1’. So, you should type 1 in the myid file and save it under the node data folder. And add a cluster node endpoint ‘server.1=server id: port1: port 2’ in the configuration file as well.
9. Start the Zookeeper node by using the following command                                                                          

• Integrate Zookeeper with Kafka

This is a very important step to integrate both. However, the only thing you need to do is to edit the configuration and update the Zookeeper setting for the Kafka cluster node when you want to use the custom Zookeeper service. Open the server.properties file under Kafka config folder. And then save and update the file. Now, you can restart your Kafka cluster to use your custom Zookeeper service.