What is Kafka? Apache Kafka explained with core concepts and a Node.js example

So what is Kafka? Apache Kafka is an open source distributed event streaming platform built by LinkedIn and now managed by the Apache Software Foundation. It moves real-time data between systems at massive scale, and it powers everything from Uber’s ride matching to Netflix’s recommendation pipeline. This guide covers how it works, its core parts and how to use it from Node.js.

What is Kafka

Kafka is a distributed system that lets applications publish, store and process streams of events in real time. An event is just a record of something that happened, such as a payment, a page view or a sensor reading.

Kafka runs as a cluster of servers called brokers. Producers write events to named categories called topics, and consumers read from those topics independently. Unlike a normal message queue, Kafka does not delete a message once it is read. It keeps events on disk for a retention period you configure, so multiple consumers can read the same stream at their own pace.

This design gives Kafka three properties that matter for real-time systems: high throughput, low latency and fault tolerance through replication across brokers.

A brief history of Kafka

LinkedIn built Kafka in 2011 to solve a real problem: its existing systems could not track user activity fast enough as traffic grew. The team needed something that combined messaging, storage and processing in one tool, so they built Kafka and donated it to the Apache Software Foundation.

The name comes from author Franz Kafka. Jay Kreps, one of the original creators, picked it simply because he liked Kafka’s writing and thought a “system optimized for writing” deserved the name.

Several of Kafka’s original authors went on to found Confluent, a company that builds commercial tooling and a managed cloud service around the open source project.

Core components of Kafka

Every Kafka deployment is built from the same handful of parts:

• Broker: a Kafka server that stores data and serves client requests. A cluster typically runs several brokers together.
• Topic: a named stream of events, similar to a folder. Producers write to topics and consumers read from them.
• Partition: a topic is split into partitions so data can spread across brokers. Order is guaranteed within a partition, not across the whole topic.
• Producer: an application that publishes events to a topic.
• Consumer: an application that subscribes to a topic and reads events from it.
• Consumer group: a set of consumers that split the work of reading a topic so no message is processed twice within the group.
• Offset: a sequential ID for each event within a partition. Consumers track their offset to know where they left off.

How Kafka works

Kafka’s flow of data breaks down into four steps.

Producers publish events to a topic

A producer app sends events (logs, transactions, clicks) to a Kafka topic. Kafka appends each event to the end of a partition’s log, in the order it arrives.

Topics split across partitions and brokers

Each topic is divided into partitions, and those partitions are distributed across the brokers in the cluster. This is what lets Kafka scale horizontally instead of relying on one machine’s disk and network.

Consumers read using offsets

Consumers pull events from partitions using their tracked offset. This means a consumer can read from the latest event, replay from the beginning or resume exactly where it stopped after a restart.

Brokers replicate partitions for durability

Every partition has a leader broker and one or more follower replicas. If the leader fails, a follower takes over automatically, so no data is lost and reads and writes continue. If you want hands-on practice with this, our guide on setting up a multi-broker Kafka cluster on AWS EC2 walks through building one from scratch.

Kafka APIs

Kafka exposes four core APIs for building on top of it:

Most application code only touches the Producer and Consumer APIs directly. Streams and Connect are typically used for building pipelines rather than one-off apps.

Using Kafka with Node.js

The most common Node.js client for Kafka is kafkajs. Install it first.

npm install kafkajs

Here is a minimal producer that connects to a local broker and sends one event to a topic called orders.

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['localhost:9092']
})

const producer = kafka.producer()

async function sendOrder() {
  await producer.connect()
  await producer.send({
    topic: 'orders',
    messages: [
      { key: 'order-101', value: JSON.stringify({ item: 'Timex watch', qty: 1 }) }
    ]
  })
  await producer.disconnect()
}

sendOrder()

And here is a consumer that reads from the same topic as part of a consumer group.

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'order-processor',
  brokers: ['localhost:9092']
})

const consumer = kafka.consumer({ groupId: 'order-processors' })

async function run() {
  await consumer.connect()
  await consumer.subscribe({ topic: 'orders', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      console.log(`Partition ${partition}: ${message.value.toString()}`)
    }
  })
}

run()

Notice groupId in the code above. Run multiple instances of this consumer with the same group ID and Kafka splits the topic’s partitions between them automatically, giving you parallel processing for free.

Why Kafka needs coordination: Zookeeper and KRaft

For most of its life, Kafka depended on Apache Zookeeper to track broker metadata, elect partition leaders and detect failures. Zookeeper worked, but it meant running and maintaining a second distributed system just to keep Kafka running.

In 2021, Kafka introduced KRaft (Kafka Raft), which moves this coordination into Kafka’s own brokers using the Raft consensus protocol. New Kafka clusters no longer need Zookeeper at all, which cuts operational overhead and simplifies deployment. If you’re still running an older Kafka version that requires it, see our guide on setting up a Zookeeper cluster for Kafka on AWS EC2. If you are starting a new project today, use KRaft mode instead.

Kafka vs RabbitMQ

Kafka and RabbitMQ solve similar problems but with different trade-offs.

If you need to replay historical events or fan a single stream out to several independent readers, Kafka fits better. If you need flexible routing logic and simple point-to-point delivery, RabbitMQ is usually simpler to run.

Tools that work with Kafka

Kafka rarely works alone. It sits at the center of a larger data stack, and a few tools show up next to it in most production setups. Knowing what each one is for helps you avoid reinventing something Kafka’s community has already built.

• Kafka Connect: ships with Kafka and handles moving data in and out through pre-built connectors for databases, S3, Elasticsearch and more, without custom code.
• Kafka Streams: a Java library for building stream-processing apps directly on top of Kafka, useful for filtering, joining and aggregating events without a separate cluster.
• Apache Spark and Apache Flink: both read from Kafka topics to run heavier analytics, machine learning pipelines or windowed aggregations that go beyond what Kafka Streams handles alone.
• Schema registries: tools like Confluent Schema Registry validate that producers and consumers agree on message structure, which matters once more than one team writes to the same topic.

You rarely need all of these on day one. Most teams start with just producers, consumers and a handful of topics, then add Connect or a stream-processing layer once the data volume or the number of consuming teams grows.

Common use cases

Kafka shows up anywhere data needs to move between systems fast and reliably. Some of the most common patterns:

• Real-time analytics: tracking page views, app events or IoT sensor data as it happens.
• Microservices communication: services publish events instead of calling each other directly, which keeps them decoupled. Codeforgeek’s own microservices architecture with Node.js guide covers this pattern in more depth.
• Log aggregation: collecting logs from many servers into one durable, queryable stream.
• Event sourcing: storing every state change as an ordered event instead of overwriting a row in a database.
• Fraud and anomaly detection: scanning transaction streams for suspicious patterns in real time.

Companies like Uber, Netflix, LinkedIn and Airbnb all run Kafka at the core of systems that cannot tolerate lag or data loss. What connects these use cases is scale: each one involves a constant, high-volume stream of events rather than occasional, one-off messages.

Benefits and challenges

Kafka’s main strengths are throughput, durability and horizontal scalability. A well-tuned cluster handles millions of messages per second with latency in the single-digit milliseconds, and replication means a broker failure does not mean data loss. Because storage is cheap and retention is configurable, you can also replay days or weeks of history through a new consumer without touching the original producers.

The trade-off is operational weight. Running Kafka well means tracking partition counts, replication factors, retention settings and consumer lag. Rebalancing a cluster when you add brokers takes planning, and a misconfigured retention policy can quietly fill up disks. Small projects with low message volume are often better served by a simpler queue or even a Postgres table, since Kafka’s benefits show up mainly at scale.

Key takeaways

• Kafka is a distributed event streaming platform built by LinkedIn and open sourced through Apache.
• Producers write events to topics, consumers read them and brokers store and replicate the data.
• Topics are split into partitions to scale across multiple brokers.
• Offsets let consumers track their read position and resume after a restart or failure.
• KRaft mode removes the need for a separate Zookeeper cluster.
• Kafka keeps messages after they’re read, unlike traditional queues such as RabbitMQ.
• kafkajs is the standard way to produce and consume Kafka events from Node.js.
• Kafka fits high-throughput, replayable event streams, not simple point-to-point task queues.