Setting up Kafka management for kafka cluster

So far we have set up the Kafka cluster with multiple broker and multiple zookeepers. We need a way to see our configuration in a presentable manner.

In this article, we are going to set up the Kafka management software to manage and overview our cluster.

The software we are going to use is called Kafka manager developed by folks at Yahoo.

Installing Kafka manager

You need to have Java v8 installed in your system before installing the Kafka manager.

Download the Kafka manager from Github.

git clone https://github.com/yahoo/kafka-manager

wget https://github.com/yahoo/kafka-manager/archive/1.3.3.22.zip

If you have downloaded the zip file, unzip it using the unzip utility of Ubuntu.

Once the download is completed, we need to build the Kafka manager.

Switch to the Kafka manager folder and run this command.

$ cd kafka-manager
$ ./sbt clean dist

Then go ahead and grab a coffee. This will take some time.

Once the build is completed, you should be seeing a target directory created inside the Kafka manager folder.

Before we start the Kafka manager, we need to do a configuration.

Open up application.conf file which is in the conf directory.

$ vi conf/application.conf

Change the following values:

kafka-manager.zkhosts="localhost:2181"
basicAuthentication.username="admin"
basicAuthentication.password="password"

This is a basic configuration to get you started.
Now run the Kafka manager. First switch to this directory.

$ cd target/universal

You should be seeing a zip file created which is our build. Unzip it and then switch to the folder.

$ cd kafka-manager-1.3.3.22

Now run the Kafka manager using the following command.

$ bin/kafka-manager

By default, it runs on port 9000. Open Kafka manager from your local machine by typing

:9000

Type in the username and password you have set in the config. You should be seeing a Kafka manager screen.

From the Cluster menu, click on Add Cluster button. Give it a proper name and type the zookeeper host addresses separated by commas.

In our case, we have 3 zookeepers running on 3 EC2 instances. So add all three addresses like this.

:2181,:2181,:2181

Check the Enable JMX Polling button and click on save. Kafka manager will register the cluster and provide you all information about it.

Now you should be seeing cluster information like this.

You can view topics, brokers and their profiling information using Kafka manager.

You can create, list topics too.

Kafka manager is really simple and easy to set up a tool for our Kafka cluster.

This article is a part of a series, check out other articles here: