Kafka

How many brokers should we have? What is the ideal ram size? Should we use RAID or SSD? We come across questions like this while configuring and deploying a Kafka cluster. In this article, we will cover in brief about the Kafka capacity estimation and planning. Please note that this is subjective advice and you […]

So far, we have set up a Kafka cluster with an optimal configuration. It’s time to do performance testing before asking developers to start the testing. In order to do performance testing or benchmarking Kafka cluster, we need to consider the two aspects: Performance at Producer End Performance at Consumer End We need to do

So far we have set up the Kafka cluster with multiple broker and multiple zookeepers. We need a way to see our configuration in a presentable manner. In this article, we are going to set up the Kafka management software to manage and overview our cluster. The software we are going to use is called

In the previous article, we have set up the Zookeeper and Kafka cluster and we can produce and consume messages. In this article, we will do the authentication of Kafka and Zookeeper so if anyone wants to connect to our cluster must provide some sort of credential. We have 3 Virtual machines running on Amazon

Apache Kafka is a distributed streaming platform used by 100’s of companies in the production environment. Kafka, in a nutshell, allows us to do three important tasks: Publish and subscribe to streams of data. Store streams of data. Process streams of data. With these capabilities, we can use Kafka in a various situation such as

Zookeeper is a centralized service to handle distributed synchronization. Kafka uses zookeeper to handle multiple brokers to ensure higher availability and failover handling. But what if zookeeper failed? We can’t take a chance to run a single Zookeeper to handle distributed system and then have a single point of failure. To handle this, we run

In this article, we are going to learn the basics of Apache Kafka and it’s core concepts. And yes, a bit of history too. What is Kafka Apache Kafka is an open source distributed streaming platform developed by LinkedIn and managed by the Apache software foundation. Kafka is aimed to provide a high-throughput, low-latency, scalable,