Apache Arrow

The problem Arrow addresses

Consider a fairly ordinary pipeline. A file is read, loaded into a data frame library, passed to a query engine, and the result is sent to another process for a calculation.

Each of those components has its own way of holding data in memory. So at every boundary the data is converted: written out in some intermediate form, parsed back in and rebuilt in the next component's structures.

That conversion work is frequently a large share of the total time. It produces nothing. The data is the same before and after.

What Arrow does about it

Apache Arrow defines a standard layout for columnar data in memory. Not a file format for storage, but a specification of how values sit in memory while a program is working with them.

When two components both use that layout, the conversion step disappears. Data passes between them as it already is. This is what the documentation means by zero copy: the receiving component reads the existing bytes rather than parsing a copy.

Columnar, and why

Arrow stores values column by column rather than row by row.

Analytical work reads columns. Summing a value across a million records touches one column and ignores the rest. When that column's values sit together in memory, the processor reads them efficiently. When they are scattered between other fields of each row, it does not.

The layout also records null values separately, so missing data is handled consistently across every implementation rather than each library inventing its own convention.

More than a specification

The project provides implementations of the format for many languages, including C plus plus, Python, Java, Go, Rust and R. Because they all follow the same specification, a component written in one can hand data to a component written in another.

Around the core format sit several related pieces:

• A file and streaming format for writing Arrow data out or sending it over a connection.
• A transport protocol for moving Arrow data between systems over a network.
• A database connectivity interface for retrieving query results in Arrow form directly.
• Compute functions for operating on Arrow data in place.

Where you encounter it

Most people use Arrow without setting out to. Data frame libraries, query engines and file readers increasingly use it internally, so the conversion savings arrive without being requested.

You work with it directly when building something that passes data between components, or when you want a query result handed to another tool without a round trip through a row based format.

Who it is for

Arrow is aimed at people building data tools and pipelines rather than at end users. Data engineers, library authors and platform teams are the ones who choose it deliberately.

Points to keep in mind

Arrow is not storage. Data held in Arrow format lives in memory, and something else is needed to persist it, which is usually a columnar file format on disk.

The benefit also depends on both sides participating. Handing Arrow data to a component that does not understand it means converting anyway, so the saving only appears where the format is shared.

Getting started

The documentation covers the columnar specification itself along with a guide for each language implementation. Reading a file into an Arrow table in one of those languages and inspecting its schema is the usual introduction.