Apache Beam

What is Apache Beam?

Apache Beam is a unified model for defining both batch and streaming data parallel processing pipelines. It is a project of the Apache Software Foundation.

Beam separates two things that are usually joined together. The description of the work is written in a Beam SDK. The execution of that work is done by a runner, which translates the pipeline for a specific processing system.

Why Apache Beam exists

Batch and streaming are often written twice, once for files that have ended and once for events that keep arriving. Beam expresses both with the same set of concepts, so one pipeline can handle bounded datasets and unbounded streams.

The second reason is portability. Because the runner is chosen separately, the same pipeline can move to another engine without the logic being rewritten.

How a Beam pipeline executes

The model has a small vocabulary.

1. A Pipeline is the graph of transformations you build.
2. A PCollection is a dataset or stream inside the pipeline, described as an unordered bag of elements.
3. A PTransform is a processing step applied to a PCollection.
4. Windowing subdivides a PCollection into finite windows using the timestamps on its elements.
5. A watermark estimates when all the data for a window has arrived.
6. A trigger decides when the aggregated result for a window is emitted.
7. A runner executes the finished pipeline on the chosen processing system.

In practice you create a pipeline object in your SDK language, apply transforms to collections and submit the result to a runner. The transforms then process elements in parallel across workers.

Who Apache Beam is for

Beam is aimed at pipeline authors working in one of its SDK languages, and at organisations that want a single approach covering both batch and streaming.

Teams that prefer to stay independent of one processing engine are a natural fit, since the runner is a choice made at submission time rather than a decision baked into the code.

Where Apache Beam stops

Beam does not run anything by itself. It always needs a runner, and the behaviour you observe depends on the engine behind that runner.

You also have to work within the available SDKs. The documentation references Java, Python and Go quickstarts, so a team using another language would need a different approach.

Writing a first Beam pipeline

Three quickstart guides are published, one for each SDK language. The steps are the same in all of them. Create a pipeline object, define the collections that hold your data, apply the transforms that do the work and submit the pipeline to a runner.

For a first run, the Direct Runner executes the pipeline locally, which is useful for testing before you move to Apache Flink, Apache Spark or Google Cloud Dataflow.