Apache HBase

The gap it fills

Hadoop storage was built for processing files in bulk. A job reads large files, does work across them and writes results. It handles enormous volumes and it is efficient at exactly that pattern.

What it does not offer is reaching a single record. Retrieving one customer's row from a dataset spread across a cluster is not something bulk file processing does, and running a job to find it is disproportionate.

HBase adds that capability on the same storage. Individual rows can be read and written, over datasets far too large for one machine.

Rows, families and cells

A table has rows, and each row is identified by a row key.

Columns are grouped into column families. A family is a set of columns stored together physically.

Within a row, each column holds a cell, and a cell can retain several versions distinguished by timestamp.

Two aspects differ from a relational table.

Columns are sparse. A row stores only the columns it actually has. A table with a thousand possible columns where each row uses ten costs storage for the ten. There are no empty cells taking up space, which is what makes very wide sparse tables practical.

Column families are a physical decision rather than a naming convention. Compression settings and how many versions to keep are configured per family. Because families determine layout, the documentation advises keeping their number small, and this is a design decision made early rather than adjusted casually.

Sorted order, and why the key decides everything

Rows are stored in sorted order by row key. Tables are divided into regions, each covering a contiguous span of keys, and regions are distributed across servers.

This makes the row key the most consequential design decision.

Sorted storage means keys adjacent in order are adjacent on disk. A key combining an identifier with a timestamp puts one subject's history together, so reading that history is a contiguous scan rather than scattered lookups.

The same property creates the classic pitfall. A key that increases steadily, a plain timestamp being the obvious example, means every new write lands at the end of the key space and therefore in one region on one server. That server saturates while the rest of the cluster does nothing.

The documented remedies involve distributing keys deliberately, by prefixing or by hashing part of the key, so that writes spread while related records still group usefully.

Everything about performance here comes back to this decision.

Versions

A cell can keep several versions of its value, each with a timestamp, with the number retained configured per family.

This means the history of a value is available without building a separate history table. Reading normally returns the most recent version, and older ones can be requested.

Whether this is useful depends entirely on the application. Where the previous state of a value matters, it removes work. Where it does not, retaining versions is storage spent for nothing, which is why the setting exists.

What comes with it

HBase depends on other components. It stores data on the Hadoop Distributed File System and uses a coordination service to track cluster state.

This is worth stating plainly because it sets expectations about operations. Running HBase means running the supporting components too, and the operational burden covers all of them rather than HBase alone.

Where a Hadoop environment already exists, this is a small addition. Where it does not, it is a substantial commitment for what would otherwise be a database choice.

Who uses it

HBase is used by teams operating large Hadoop environments who need record level access to data held there. It appears in telecommunications, financial services and other settings with very large volumes of records that must remain individually retrievable.

Points to consider

HBase is an Apache project under an open source licence, and the official reference guide is the documentation for it.

Its natural setting is an existing Hadoop environment. Adopting it independently means taking on the surrounding components as well.

Row key design cannot be treated as an afterthought. It determines both write distribution and which reads are efficient, and changing it after data exists means rewriting the table.

It is not a relational database and does not attempt to be. There are no joins and no SQL in the core, though separate projects provide SQL layers over it. Requirements involving flexible querying across entities are not what it addresses.

Getting started

The official reference guide covers the data model, row key design, regions and operations. The sections on key design are the ones to read first, since they determine whether a deployment performs sensibly and they are difficult to revisit later.