Apache Atlas

What Atlas is for

Apache Atlas records metadata about data assets and governs them: what exists, how assets relate, and what markings apply to them.

It comes from the Hadoop ecosystem, and its integrations reflect that origin. Understanding where it fits best starts from knowing that.

Lineage captured rather than documented

The capability that most justifies Atlas is how lineage arrives.

Lineage can be documented. Somebody writes down that this table is built from those two. That description is accurate on the day it is written and progressively less so afterwards, because pipelines change and documentation does not.

Atlas takes a different route. Hooks in the processing engines report activity as it happens. When a job runs, the engine says what it read and what it wrote.

The lineage that results describes what actually ran. It does not go stale, because it is regenerated by execution rather than maintained by hand.

Classifications

Assets can carry classifications: labels marking them as containing personal data, as restricted, as belonging to a particular domain.

On its own that is documentation. Two things make it more than that.

Propagation along lineage

Classifications can follow lineage.

This addresses a real gap in data governance. A source table is correctly marked as containing personal data. A pipeline derives an aggregate from it, then another table from that. Those derived tables contain information originating in personal data, and nobody marked them, because marking happens at the source.

Propagation means the derived assets inherit the markings. Sensitivity is not lost through transformation, which is exactly where it is usually lost.

Classification driving access control

The second is that classifications can be consumed by an authorisation system.

A policy written against a classification rather than against a list of tables covers every asset carrying that label, including ones created after the policy was written.

That is the difference between governance that scales and governance that requires somebody to remember to update a list.

The type system

What Atlas can record is defined by types, and the type system is extensible.

That matters because every platform has components nobody anticipated. Rather than those being invisible to the catalogue, a type is defined for them and they appear alongside everything else.

Notifications

Metadata changes are published as messages, so other systems can react. That is what allows an authorisation system to respond to a new classification without polling.

Who uses Atlas

Atlas is used on Hadoop based data platforms, particularly in regulated industries where recording lineage and tracking sensitive data are requirements rather than conveniences.

Points to consider

Its integrations are strongest in the ecosystem it came from. Outside that, how useful it is depends on what integrations exist for your systems, and that is worth checking before adoption.

It is also a system to operate, with its own dependencies to deploy and maintain.

Lineage depends on the hooks being installed and configured. A processing engine without one is invisible, so lineage that appears complete may have gaps, and knowing where those are matters.

Getting started

The documentation covers the type system, installing hooks for supported engines, classifications and their propagation, and the interfaces available. Enabling a hook on one engine and watching lineage appear as jobs run demonstrates the central mechanism.