Apache Flink
A framework for processing continuous data streams with local state, event time and exactly once recovery.
Apache Flink is a stream processing framework. It reads data as a continuous flow of events rather than as finished files, keeps state as it goes and produces results that stay consistent when a machine fails. The documentation describes Flink as a framework for distributed stream processing that runs in parallel across a cluster.
What is Apache Flink?
Apache Flink is a framework for distributed stream processing. A stream is a flow of data with no fixed end, such as clicks on a website or readings from a sensor. Flink processes that flow as the events arrive instead of waiting for a file to be complete.
Flink runs across a cluster. A job is split into operators, and each operator runs as several subtasks in parallel, each one working on its own partition of the stream.
What Apache Flink is used for
The documentation lists three areas: streaming ETL, analytics and event driven applications. Two examples it gives are counting events for a dashboard that updates every minute, and computing the features used in fraud detection.
Flink handles finished datasets too. A stream may be unbounded, meaning it has no defined end, or bounded, meaning it does.
How Apache Flink processes a stream
- A source reads records into the job.
- Operators apply your logic to each record and pass results along the stream.
- Operators that need memory keep state locally, described in the documentation as a sharded key value store where each instance holds a set of key groups.
- Timestamps carried in the data drive event time processing, so results depend on when an event happened rather than when it was handled.
- Snapshots of the whole pipeline state, including input offsets, are taken without pausing the job. After a failure, Flink restores from a snapshot.
- A sink writes the results out.
Who Apache Flink suits
Flink fits organisations that work with continuous data and need results they can rely on. The documentation points to systems that require fault tolerant distributed processing, and to cases where the same answer is expected from historic data and from live data.
Three APIs are available, so a team can pick the level it is comfortable with: Flink SQL, the Table API and the DataStream API.
Points to be aware of
Flink expects distributed cluster infrastructure. It is not a library you add to a single application and forget about.
State can also grow large. The documentation notes that this may call for disk based data structures rather than memory alone.
Ordering is not guaranteed everywhere. Where a stream is redistributed between operators, the order in which results arrive is non deterministic by design.
First steps with Apache Flink
The documentation provides tutorials for each API, along with a local installation guide and an operations playground.
Flink SQL is the shortest path if you already know SQL. The DataStream API gives the most control over state and time, so it is the one to reach for when the logic depends on either.
Key features of Apache Flink
Capabilities described in the official documentation.
Stateful processing
Flink manages state across distributed operators and keeps it local to each subtask for fast access.
Event time processing
Records are handled using timestamps carried in the data, which makes results reproducible.
Exactly once semantics
State snapshots let Flink recover after a failure without counting the same record twice.
Parallel execution
A job runs as independent operator subtasks spread across the machines in a cluster.
Advantages of Apache Flink
Factual advantages that follow from the features above.
Consistent results across live and historic data
Because processing uses event timestamps, replaying old data produces the same answers as processing it live.
Recovery without losing position
Snapshots record input offsets alongside state, so a restarted job resumes from a known point.
A choice of API level
Flink SQL, the Table API and the DataStream API give different levels of control over the same engine.
Late arrivals are handled deliberately
Event time processing lets a record that arrives out of order still be counted in the period it belongs to.
Common use cases for Apache Flink
Situations the official documentation describes this tool as being used for.
Streaming ETL
Records are read, cleaned and written to a target continuously rather than in a scheduled batch.
Counting events per minute
A dashboard is fed by counts computed over short windows as the events arrive.
Feature computation for fraud detection
Values needed by a fraud model are computed from the live event stream.
Joining two live streams
Two continuous inputs are joined on a key within a time boundary so related records are matched as they arrive.
Official website
Everything on this page is based on the official documentation for Apache Flink. You can read the source here.
Apache Flink official documentation