Apache Oozie

What Oozie is for

Apache Oozie is a workflow scheduler for Hadoop. Its purpose is to run a series of jobs in the right order, at the right time, on a Hadoop cluster.

Processing on a cluster rarely consists of one job. Data is ingested, then transformed, then aggregated, then loaded, and each step depends on the one before it. Oozie exists to describe that sequence and run it.

Workflows

A workflow is the sequence itself. It is written as an XML document listing actions and the transitions between them.

An action is one job. The available types reflect where Oozie came from, covering the job types a Hadoop platform runs, along with shell commands, notifications and calling another workflow.

Between actions sit control nodes that shape the path:

• Decision nodes branch depending on a condition.
• Fork and join nodes run several actions at the same time and wait for all of them to finish.
• Kill nodes end a workflow when something has gone wrong.

Coordinators, and the idea worth taking away

A workflow describes what runs. A coordinator decides when.

The obvious answer to when is a schedule. Every night at two. Oozie supports that, and so does every other scheduler.

Its more interesting capability is waiting for data. A coordinator can be told which input a workflow depends on, and hold the run back until that input exists.

This addresses a failure that is very common in scheduled pipelines. A job is set to run at two because the upstream feed usually lands by half past one. On the night the feed is late, the job runs anyway, finds nothing and either fails or, worse, succeeds against incomplete data.

With a data availability trigger, the run waits. The dependency is stated rather than approximated by a clock.

Bundles

Coordinators can be grouped so that a set of related recurring jobs is managed together rather than individually. This matters on platforms where the number of scheduled jobs grows into the hundreds.

How work is submitted

1. The workflow definition and its supporting files are placed in cluster storage.
2. A properties file supplies the values the workflow needs.
3. The job is submitted to the Oozie server.
4. The server runs the actions in order, launching each on the cluster.
5. Progress and history are visible through the server's interface.

Who used it

Oozie is found on established Hadoop platforms, operated by the data engineering or platform team responsible for scheduled processing. It is most often encountered today by teams working with an existing estate rather than starting a new one.

Points to be aware of

XML is verbose. A workflow with a dozen actions is a long document, and this is the most common complaint about Oozie compared with schedulers where workflows are written in a programming language.

Its action types are also oriented towards Hadoop. That is efficient inside such a platform and limiting outside one.

As with any long established project, checking the official site for release activity and current status is worth doing before selecting it for new work.

Reference

The official project site publishes the documentation, covering the workflow specification, every action type, the coordinator and bundle specifications and the command line interface used to submit and manage jobs.