Watching Kafka work: batches, log segments and an idle fourth consumer

Most explanations of Kafka stay at the diagram level: producers, a log, partitions, consumers. I wanted to see those ideas in the files and tools Kafka ships with. So I started one broker, wrote 3,000 keyed messages into a topic with three partitions, and looked at what ended up on disk and who got which partition.

Everything below ran in the official apache/kafka:4.3.1 image on a laptop. Offsets and batch sizes will differ on your machine, but the shapes should match.

What I found, in one table

Question	What the tools showed
Messages per partition	900, 1,080 and 1,020 (3,000 in total)
Files behind partition 0	one `.log` segment, an `.index`, a `.timeindex`
Batches for 900 messages	2 (670 records, then 230)
Where key `user-7` went	partition 2, all 60 messages
4 consumers, 3 partitions	the fourth consumer got 0 partitions
After a group read everything	earliest offset still 0 in every partition
Default retention	`retention.ms=604800000` (7 days), `cleanup.policy=delete`

Setup

One broker in KRaft mode, no ZooKeeper. Kafka 4.0 removed ZooKeeper completely, so the official image starts a combined broker and controller by default.

docker run -d --name kafka apache/kafka:4.3.1
docker exec -it kafka bash
cd /opt/kafka/bin

./kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

Created topic orders.output

The 3,000 messages use 50 different keys, user-0 to user-49, so every key appears 60 times:

for i in $(seq 1 3000); do echo "user-$((i % 50)):order $i"; done | \
  ./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
  --reader-property parse.key=true --reader-property key.separator=:

./kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders

orders:0:900
orders:1:1080
orders:2:1020output

The split is uneven because Kafka does not spread messages, it spreads keys. 15 keys hash to partition 0, 18 to partition 1 and 17 to partition 2, and each key brings its 60 messages along.

A partition is a folder of append-only files

ls -la /tmp/kraft-combined-logs/orders-0

-rw-r--r-- 1 appuser appuser 10485760 Oct 18 17:10 00000000000000000000.index
-rw-r--r-- 1 appuser appuser    22103 Oct 18 17:10 00000000000000000000.log
-rw-r--r-- 1 appuser appuser 10485756 Oct 18 17:10 00000000000000000000.timeindex
-rw-r--r-- 1 appuser appuser        8 Oct 18 17:10 leader-epoch-checkpoint
-rw-r--r-- 1 appuser appuser       43 Oct 18 17:10 partition.metadataoutput

The .log file is the data: 900 messages in 22 KB. Its name is the offset of the first message in it. When a segment reaches its size limit, Kafka closes it and starts a new file named after the next offset, so old segments are never rewritten. The .index and .timeindex files map offsets and timestamps to byte positions in the log, which is how a consumer can jump to offset 600 without reading the first 599 messages. Both are preallocated at 10 MB and trimmed when the segment is closed, which is why they look so large next to a 22 KB log.

900 messages, 2 writes

kafka-dump-log.sh prints the record batches inside a segment:

./kafka-dump-log.sh --files /tmp/kraft-combined-logs/orders-0/00000000000000000000.log

Dumping /tmp/kraft-combined-logs/orders-0/00000000000000000000.log
Log starting offset: 0
baseOffset: 0 lastOffset: 669 count: 670 ... size: 16371 magic: 2 compresscodec: none ...
baseOffset: 670 lastOffset: 899 count: 230 ... position: 16371 ... size: 5732 ...output

The producer did not send 900 requests for partition 0. It collected messages in memory and sent them as two batches, one with 670 records and one with 230. The broker appends each batch to the end of the file as it is, and a consumer later reads the same batch back. That is the whole point of batching: one network round trip and one disk write for hundreds of messages. compresscodec: none also tells you where compression would happen; with compression.type set on the producer, the batch is compressed once and stays compressed on disk and on the wire.

The same key always lands in the same partition

for p in 0 1 2; do
  n=$(./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
      --partition $p --offset earliest --property print.key=true --timeout-ms 6000 \
      | cut -f1 | grep -cx user-7)
  echo "partition $p: $n"
done

partition 0: 0
partition 1: 0
partition 2: 60output

All 60 messages for user-7 are in partition 2, in the order they were sent. This is the ordering guarantee Kafka gives you: per key, inside one partition, not across the topic. It is also why adding partitions to a live topic is not free. The hash of a key is taken modulo the partition count, so with a different count some keys move to a different partition, and their old and new messages are no longer in one ordered log.

Four consumers, three partitions

I started four console consumers in the same group, billing, and asked the group who reads what:

for c in 1 2 3 4; do
  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
    --group billing --consumer-property client.id=consumer-$c > /dev/null &
done
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing --members --verbose

GROUP     CLIENT-ID    #PARTITIONS  CURRENT-ASSIGNMENT
billing   consumer-1   1            orders:0
billing   consumer-3   1            orders:2
billing   consumer-2   1            orders:1
billing   consumer-4   0            -output

Inside a consumer group, each partition goes to exactly one consumer. With three partitions, the fourth consumer is connected, healthy and idle. The partition count is the ceiling for parallelism in a classic consumer group, so if you expect to run ten consumers, the topic needs at least ten partitions before you need them.

Kafka 4.2 made share groups production ready, which is the other answer to this limit: in a share group, several consumers can read from the same partition and acknowledge records one by one, closer to how a queue works. You give up per-key ordering in exchange.

Reading does not delete anything

The billing group has read every message. A new group starts from the beginning and gets all of them again:

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
  --group analytics --from-beginning --timeout-ms 10000 | wc -l
./kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders --time -2

3000
orders:0:0
orders:1:0
orders:2:0output

--time -2 asks for the earliest offset still stored, and it is 0 in every partition. A consumer’s position is only an offset that the group stores; the messages stay until retention removes them:

./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics \
  --entity-name orders --describe --all | grep -E " retention.ms=| cleanup.policy="

  cleanup.policy=delete ...
  retention.ms=604800000 ...output

604,800,000 ms is seven days. Retention deletes whole closed segments, which is cheap because it is a file delete, not a scan. With cleanup.policy=compact, Kafka instead keeps the latest record for every key, which turns a topic into a changelog of current values.

Why this is fast

None of the steps above needed special hardware. The speed comes from a few choices that work together:

Producers send batches, so the cost of a request is shared by hundreds of records.
The broker only appends to the end of a segment file, which is the access pattern disks and SSDs handle best.
Kafka leaves caching to the operating system’s page cache, so a consumer that is close to the end of the log usually reads from memory.
When a consumer fetches data, the broker can hand the file bytes to the network socket without copying them through the JVM (zero-copy).

For scale: LinkedIn reported in 2019 that it was sending more than 7 trillion messages a day through Kafka.

Clean up

docker rm -f kafka