AWS Glue
A serverless AWS service that catalogues your data and runs the jobs that prepare it, without a cluster to manage.
AWS Glue is a serverless data integration service. It has two main halves. The Data Catalog records what data you have and where it is, and the job service runs the code that reads, transforms and writes that data. Because it is serverless, AWS provisions the compute for each run rather than you keeping a cluster running.
AWS Glue in short
AWS Glue is a data integration service on AWS. The developer guide describes it as serverless, which is the detail that shapes how it is used: there is no cluster to create, size or shut down. You define the work and AWS runs it.
Glue is really two things that are often discussed as one. Understanding them separately makes the service much easier to follow.
Part one: the Data Catalog
The Data Catalog is a record of what data exists. It holds databases, tables, columns, types and the location of the underlying files.
It does not hold the data itself. A table in the catalog is a description that points at files sitting in storage.
This matters because the catalog is shared. Other AWS analytics services can read the same definitions, so a table registered once is available to more than one query engine without being described again for each.
Part two: jobs
A job is the work that reads, transforms and writes data. Glue jobs can be written directly or built on a visual canvas that generates the script for you.
When a job runs, AWS provisions the compute for that run and releases it afterwards. Nothing sits idle between executions, which is a different cost model from keeping a cluster available.
Jobs can be triggered on a schedule, by an event or on demand, and can be chained so one starts when another finishes.
Where crawlers fit
A crawler is how the catalog gets populated when you do not already know what is in your storage.
You point a crawler at a location. It samples the files, works out their format and columns, and creates or updates table definitions from what it finds. Running it again picks up new partitions and changed structures.
If the schema is already known, defining tables directly gives more control, particularly over column names and types where inference might guess differently from what you intended.
A typical arrangement
- Files arrive in object storage.
- A crawler registers or updates the table definition in the Data Catalog.
- A job reads that table, applies its transformations and writes the result to a prepared location.
- A bookmark records how far the job got so the next run does not repeat work.
- Query services read the prepared tables through the same catalog.
Who Glue is for
Glue suits teams building data pipelines on AWS who would rather not operate processing clusters. It is aimed at data engineers, since building a job means working with a transformation script or a visual equivalent of one.
Points worth knowing
Glue is an AWS service and is designed around AWS. That is an advantage inside that environment and a constraint outside it.
Serverless also does not mean configuration free. Worker types, capacity and job settings still influence how a job runs and what it costs, and those choices are documented in the developer guide.
Getting started
The developer guide includes a getting started section covering setting up permissions, creating a crawler, running a first job in the visual editor and reviewing the results. There are separate sections for the Data Catalog, job authoring, triggers and monitoring.
Key features of AWS Glue
Capabilities described in the official documentation.
The Glue Data Catalog
A central record of databases, tables and their schemas, which other AWS analytics services can query against.
Crawlers
A crawler scans files in storage, works out their structure and creates or updates table definitions in the catalog.
Serverless jobs
Jobs run without a cluster to provision, with AWS allocating the compute for the duration of each run.
Visual and code based authoring
Jobs can be built on a visual canvas or written directly, and the visual editor generates the underlying job script.
Advantages of AWS Glue
Factual advantages that follow from the features above.
Nothing running between jobs
Because compute is allocated per run, there is no idle cluster to pay for when no job is executing.
One catalogue for several services
Table definitions in the Data Catalog are used by other AWS analytics services, so schemas are not defined twice.
Schemas can be discovered
A crawler infers structure from the files themselves, which is useful when incoming data was not documented.
Jobs resume rather than repeat
Bookmarks record what a job has already processed, so a scheduled run handles new data instead of past data.
Common use cases for AWS Glue
Situations the official documentation describes this tool as being used for.
Preparing raw files for analysis
A job reads incoming files from object storage, reshapes them into a query friendly format and writes them back.
Cataloguing a data lake
Crawlers scan storage locations and register the tables they find so the contents of the lake are queryable.
Scheduled loading into a warehouse
A job extracts from a source, applies transformations and loads the result into a warehouse on a trigger or schedule.
Discovering the contents of unfamiliar storage
A crawler scans a storage location and reports the structures it finds when the contents are not documented.
Official website
Everything on this page is based on the official documentation for AWS Glue. You can read the source here.
AWS Glue official documentation