Azure Data Lake Storage

The directory that was never really there

Object storage generally has no directories. Objects have keys, keys often contain slashes, and tools display the result as folders. The hierarchy is a convention within names rather than a structure the storage knows about.

That works well enough for storing files. It causes specific problems for analytical workloads.

Azure Data Lake Storage is Azure object storage with a hierarchical namespace enabled, which makes directories real.

The difference sounds like a technicality. Its consequences are practical and they are the reason the option exists.

Renaming, and why it matters more than it should

Consider what a processing job does when it writes output.

A well behaved job writes to a temporary location and moves the result into place only when it has finished successfully. If the job fails partway, the temporary output is discarded and nothing incomplete appears where readers are looking.

The move is where the difference shows.

Without real directories, moving means copying every object to a new key and deleting the originals. For a job producing thousands of files this is slow, it costs requests, and it is not atomic. A failure during the move leaves half the output in place, which is precisely the situation the pattern was meant to avoid.

With a hierarchical namespace, renaming a directory is a metadata operation. It is quick and it either happens or does not.

This is why processing frameworks behave better against storage with real directories, and it is the most concrete argument for enabling it.

Permissions at the level people think in

The second consequence is access control.

Container level permissions are coarse. Either someone has access to the whole container or they do not.

Real directories allow permissions at directory and file level. A lake can hold data belonging to several teams, each able to reach their own area, without splitting into separate storage accounts.

That matters because the alternative shapes architecture badly. Where permissions can only be applied broadly, organisations create many separate storage accounts to achieve separation, and then have to manage them and move data between them.

The official documentation covers the access control models available and how they combine, which is worth understanding before a permission structure is designed, because retrofitting one across an established lake is unpleasant work.

The storage layer beneath Azure analytics

The role this service plays is being what other services read.

Azure analytics and processing services work against it directly. Data is written once and read by whichever service needs it, rather than being loaded separately into each.

This is the general point of a data lake and it depends on the storage being something many tools can read. A storage layer that only one engine understood would return the estate to copying data between systems.

Organising what is stored

Real directories do not remove the need for a sensible layout.

Data is still normally organised by dataset and then by date components, because analytical engines read only the directories relevant to a query. A query for one month should touch one month's data, and that only happens if the layout allows it.

File sizes matter too. Very many small files perform poorly, because each carries overhead that dominates when the file itself is tiny. Processing frameworks commonly combine small files for this reason.

These are the same considerations that apply to any data lake. The hierarchical namespace makes the operations on that structure efficient; it does not choose the structure.

Tiers

Data can be held in tiers priced according to access frequency. Data read constantly and data retained for years belong in different tiers, and the difference is significant at the volumes lakes reach.

Who uses it

Azure Data Lake Storage is used by data engineers, platform teams and analytics teams on Azure. It is the standard storage foundation for Azure analytics work, and most Azure data platforms have it underneath.

Points to consider

This is an Azure service and the official documentation is the reference for features, the relationship to blob storage, and pricing.

Enabling the hierarchical namespace is a decision made when a storage account is created, and the documentation covers the implications including which features apply.

Permission design is easier before a lake fills than after. Deciding the directory structure and who has access to what at the start avoids a difficult reorganisation later.

Cost includes operations as well as storage. Workloads performing very many small operations can find requests a meaningful component of the bill.

Getting started

The official documentation covers enabling the namespace, access control and integration with analytics services. Creating an account with the namespace enabled, writing data in a date structured layout, and querying it from an analytics service shows how the pieces fit together.