Apache Kafka

A log rather than a queue

The most useful way to understand Kafka is by what it is not.

A message queue holds messages until something reads them. Reading removes the message. It is delivered once, to one consumer, and then it is gone.

Kafka appends records to a log. Reading does not remove anything. Records stay for a configured retention period regardless of who has read them or whether anything has.

That single difference produces almost everything distinctive about the platform.

What retention makes possible

Several consumers, independently. A stream of order events can be read by a warehouse loader, a search index updater and a fraud checker. Each reads all the records at its own pace. In a queue, whichever consumed a message first would have taken it from the others.

Reading history again. A consumer can return to an earlier position and reprocess. This matters more than it first appears. When a bug is found in a consumer, it can be fixed and the affected records reprocessed. When a new system needs the history, it reads from the beginning rather than requiring the data to be assembled from somewhere else.

Consumers that were not there yet. A system built next year can read records produced today, provided retention covers the period. The producer never knew it would exist.

Buffering between different speeds. A producer writing quickly and a consumer working slowly are decoupled. Records accumulate in the log rather than overwhelming the consumer or blocking the producer.

Partitions

A topic is divided into partitions, held on different machines.

Partitions are how throughput scales. More of them means more can be written and read in parallel, because the work spreads across machines.

They also determine ordering, which is the detail that most often causes confusion.

Records are ordered within a partition. They are not ordered across a topic, because partitions are processed independently and there is no coordination between them.

So if the order of related records matters, and it usually does for records about the same entity, those records need to be in the same partition. This is controlled by the key used when producing: records sharing a key go to the same partition.

Getting this wrong produces a specific and confusing failure. Events about one customer are processed out of order, an update arrives before the creation it depends on, and the resulting state is wrong. Nothing errored, and the cause is that related records were spread across partitions.

Consumer groups

Consumers in a group divide the partitions between them, so each partition is handled by one member.

The group tracks its position in each partition. If a consumer stops, its partitions are reassigned and processing continues from the recorded position rather than starting again.

This is what makes consumers scalable and resilient. Add members to a group to process more in parallel, up to the number of partitions. Lose a member and the others take over.

The partition count therefore sets the maximum parallelism for a consumer group, which is why it is a decision worth thinking about when a topic is created.

The architectural consequence

Kafka is frequently adopted for a reason beyond its technical properties.

Systems that call each other directly form a web of connections. Each new system needing data from another adds a link, and the connections grow faster than the systems. Changing one system means finding everything that calls it.

Publishing records to topics inverts this. A system publishes what happens to it. Systems needing that information consume it. The producer does not know or care what reads it.

Adding a consumer requires no change to the producer. That property is why this shape of architecture is chosen as much as any throughput consideration.

Who uses it

Kafka is used by data engineers and platform teams connecting systems, feeding analytical platforms and recording events. It appears across retail, telecommunications, financial services and technology, and it frequently sits at the centre of an organisation's data movement.

Points to consider

Kafka is an Apache project under an open source licence, and the official documentation is the reference. Managed services are offered by several providers, which remove much of the operational work.

Operating a cluster is substantial. Storage, partition balance, retention and monitoring all require attention, which is why managed offerings are widely used.

Retention is a real decision with cost implications. Keeping records longer supports reprocessing and consumes storage.

Partition and key design determines both ordering and scalability, and it is awkward to change once a topic is in use. It deserves thought before rather than after.

Getting started

The official documentation covers the architecture, producing and consuming, and configuration. Producing records with keys and observing which partitions they land in demonstrates the ordering behaviour, which is the aspect that most often surprises people later.