AWS Lake Formation

Permissions against the wrong thing

A data lake is files in object storage. Access to those files is controlled by storage permissions, which apply to buckets and prefixes.

That is a mismatch with what people actually need to express.

Requirements are about tables and columns. Analysts may query the transactions table but not the columns holding customer identifiers. The team in one region may see records for that region only.

Neither of those translates into a storage permission. A prefix contains whatever files are under it, and a file contains all of its columns. Granting access to a path grants access to everything in it.

The usual workaround is producing filtered copies: one version of the table with sensitive columns removed, another restricted to one region. Each copy is another dataset to maintain, keep current and secure, and each drifts from the original.

Lake Formation moves permissions to the level requirements are expressed at.

Granting on catalogue objects

Permissions are granted against databases, tables and columns registered in the catalogue rather than against storage locations.

Analytical services honour those grants when querying registered data.

This means a grant can say what the requirement says. This principal may query this table, excluding these columns. That is one statement rather than a derived copy.

Row and cell level filtering extends it further. Filters restrict which rows a principal sees and which fields within them, so one table serves several audiences with different entitlements.

The practical effect is fewer copies. A single table with policies applied replaces the set of filtered extracts that would otherwise exist, which removes both the maintenance and the divergence.

Tags, and why individual grants do not scale

Granting on individual tables works until there are a lot of tables.

A lake accumulates them continuously. Every new table needs grants for everyone who should see it, and every new person needs grants for every table they should reach. The number of grants grows as the product of both, and maintaining it becomes a job in itself.

Worse, the failure mode is silent. A new table created without grants is invisible to people who should have it, which produces requests. A new table granted too broadly is visible to people who should not, which produces nothing at all until someone notices.

Tag based access control addresses this. Tags are applied to catalogue objects, and permissions are granted against tags.

A rule stating that a particular role may query objects tagged as finance covers every such object, including ones created next month.

Permission management then scales with the number of classifications rather than the number of tables, which is a far smaller and more stable number.

This does depend on tagging being applied consistently. The mechanism only reaches data that is labelled, so the classification practice matters as much as the permission model.

The route that bypasses the catalogue

One thing deserves emphasis because it determines whether the controls are real.

These permissions apply when data is accessed through services that integrate with the catalogue. Something reading the underlying storage directly is governed by storage permissions instead.

So the storage layer still needs appropriate controls. The point is not that storage permissions become irrelevant, but that they should permit the analytical services and not provide a general path around the finer grained rules.

The official documentation covers which services integrate and how, and understanding that is necessary rather than optional when designing an arrangement that actually holds.

Who uses it

Lake Formation is used by data platform teams, security functions and governance teams operating data lakes on AWS. It matters most where a lake holds data with genuinely different entitlements and where the alternative is producing filtered copies.

Points to consider

Lake Formation is an AWS service and the official documentation is the reference for supported services, capabilities and integration behaviour.

Coverage depends on how data is accessed. Establishing which access routes exist, and ensuring the ones that bypass the catalogue are constrained, is part of the design.

The permission model has its own concepts and takes some learning. Time spent understanding it before granting broadly is repaid, because unpicking a permission structure applied hastily across a large lake is unpleasant.

Classification is a prerequisite for tag based control. Without consistent tagging, the mechanism that makes permissions scale does not engage.

Getting started

The official documentation covers registering data, granting permissions, tag based access control and service integration. Registering one table and granting column level access to two different principals demonstrates the model quickly and shows what the analytical services enforce.