Amazon S3

Not a filesystem

The most useful thing to establish about S3 is what it is not.

It is not a disk with folders. It stores objects, each an item of data with a key identifying it, inside containers called buckets.

Keys frequently contain slashes and therefore look like file paths, and tools display them as folders. Underneath there are no directories. The key is a single string and the apparent hierarchy is a convention.

Two consequences follow, and they explain a great deal about how data lakes are built.

Objects are written and read whole. There is no modifying part of an object in place. Appending a row to a file means writing the file again. This is why analytical data is written as immutable files that are added to a collection rather than updated.

Listing has a cost. Finding objects means listing keys, and a bucket with millions of them takes real time to list. How keys are named therefore determines how efficiently anything finds what it needs.

Naming as design

Because keys are how tools find data, naming conventions are a design decision rather than housekeeping.

The standard approach puts structure into the key: a dataset name, then date components, then the file.

This allows a query for one month to look at only the keys under that month's prefix. Analytical engines rely on this, and the practice is often called partitioning because it produces the same effect as partitioning a table.

Without a convention, every query lists and examines far more than it needs. This is the most common reason a data lake performs poorly, and it is fixed at the point of writing rather than at the point of reading.

Paying for what you actually do

Storage classes trade cost against retrieval.

Data read constantly belongs in a class priced for immediate access. Data retained for compliance and read once a year belongs in one where storage is cheaper and retrieval costs more or takes longer.

The distinction matters because the volumes involved are usually large. Years of history retained at rates priced for frequent access is an expense with no corresponding benefit.

The official documentation describes the classes and their characteristics, which change and should be read directly rather than summarised.

Rules that act on age

Lifecycle rules move objects between classes or delete them as they age.

The reason this deserves attention is that data lakes accumulate. Nothing is deleted because deleting requires deciding what is safe to remove, and that decision is always less urgent than whatever else is happening.

A rule expressing the intention, that objects move to cheaper storage after ninety days and are deleted after seven years, applies continuously without anyone remembering.

Versioning is related. It retains previous versions of an object, so an overwrite or deletion is recoverable. This is protection against the ordinary accident of a process writing the wrong thing, and versions also accumulate, which is why lifecycle rules commonly cover them too.

Why it ends up at the centre

S3 is frequently the foundation of an analytics estate, and the reason is that many services read from it.

Query engines read files there. Processing frameworks read and write there. Machine learning tools take training data from there. Table formats such as Apache Iceberg organise files there into tables with proper semantics.

The result is that data can be stored once and used by several tools, rather than being copied into each system that needs it.

That property, one copy readable by many things, is what a data lake is actually for. The storage is inexpensive; the value is in not duplicating.

Who uses it

S3 is used by application developers, data engineers, platform teams and analytics teams on AWS. Its uses run from holding application uploads to serving as the storage layer beneath an entire analytics platform.

Points to consider

S3 is an AWS service. The official documentation is the reference for storage classes, consistency behaviour, limits and pricing.

Cost has more components than storage alone. Requests, retrievals and data transfer are billed, and a workload making very many small requests can cost more in requests than in storage.

Access control deserves care. Buckets holding sensitive data need permissions that are deliberate, and AWS provides mechanisms and guidance for this that are worth reading rather than assuming defaults are appropriate.

File size affects analytical performance. Very many tiny objects perform poorly because of per object overhead, which is why processing frameworks often combine small files into larger ones.

Getting started

The official documentation covers buckets, keys, storage classes, lifecycle rules and versioning. Writing a few files under a date structured key prefix and then querying them with an analytical service shows why naming conventions matter before a lake grows large enough for it to hurt.