Apache Storm

What Apache Storm does

Apache Storm is a distributed system for real time computation. The project describes it as free and open source, and its purpose is to process streams of data that never end.

The difference from batch processing is simple to state. A batch job takes a finite set of data, processes it and finishes. A Storm topology is submitted once and keeps running, handling records as they arrive, until somebody stops it.

Spouts, bolts and tuples

Storm has three terms and they are worth learning first, because everything else is described using them.

A tuple is one record moving through the system.

A spout is a source. It reads from somewhere outside, such as a message queue, and emits tuples into the topology.

A bolt is a processing step. It receives tuples, does something with them and may emit new tuples for the next bolt.

A topology is the whole graph: spouts, bolts and the connections between them.

Groupings decide where a record goes

Each component in a topology can run as several instances at once, which is how throughput is increased. That raises a question: when a bolt emits a tuple and the next bolt has ten instances, which one receives it?

The answer is the stream grouping, and it is set on each connection:

• Shuffle grouping spreads tuples randomly across instances, which suits work where each record is independent.
• Fields grouping sends all tuples sharing a field value to the same instance, which is required when a step is counting or accumulating per key.
• All grouping sends every tuple to every instance.
• Global grouping sends everything to a single instance.

Choosing the wrong grouping produces results that look almost right, which makes this one of the more important details to get correct.

Tracking a record through the graph

A record entering a topology may pass through several bolts before it is finished with. Storm follows that path.

Each component acknowledges the tuples it completes. If a tuple has not been fully acknowledged within a timeout, the spout is told and can emit it again. This is how the system deals with a worker failing part way through.

The topology has to be written to take part in this. A bolt that never acknowledges leaves records looking incomplete, so the tracking mechanism is something a developer works with rather than something that happens invisibly.

Running a cluster

A Storm cluster has a coordinating process that distributes code and assigns work, supervisor processes on each machine that start and stop workers, and a separate coordination service that holds cluster state.

Submitting a topology hands it to the coordinator, which allocates it across the available machines.

Who uses Storm

Storm suits teams that need per record processing with low delay, particularly for alerting, enrichment and running counts. It is a developer facing framework, since a topology is written in code rather than configured.

Points to consider

Storm processes one record at a time by design. That is what gives it low delay, and it also means the throughput characteristics differ from systems that process in small batches.

Correctness under failure needs attention. Retrying a record means a step may see the same record twice, so any bolt that writes somewhere has to be written with that in mind.

The project also documents a higher level API for cases where stronger processing semantics and a more declarative style are wanted.

Getting started

The project documentation includes a tutorial that builds a simple topology and runs it locally, along with a setup guide for a cluster and reference pages covering groupings, reliability and the multiple language protocol.