Azure Data Factory

What Azure Data Factory is

Azure Data Factory is Microsoft's data integration service on Azure. The documentation describes it as a cloud service for creating workflows that orchestrate data movement and transform data at scale.

Two words in that description carry the weight. It moves data, and it orchestrates. Many pipelines use both: copy something, then run something else that depends on it, then decide what to do based on the outcome.

The building blocks

Data Factory has a small vocabulary, and everything else is built from it.

Linked service. How to connect to a system. Where it is, and how to authenticate to it.

Dataset. What to read or write within that system. A table, a file, a folder.

Activity. One step. Copying data is an activity. So is running a notebook, calling a stored procedure or executing another pipeline.

Pipeline. A group of activities with an order and, where needed, conditions and loops between them.

Trigger. What starts a pipeline. A schedule, a tumbling window or an event such as a file arriving.

Integration runtime. Where activities actually execute.

The integration runtime is the piece people miss

The first four concepts are familiar from most integration tools. The integration runtime is the one worth understanding properly, because it answers a question that comes up in almost every real project: how does a cloud service read a database that sits inside a company network and is not reachable from the internet?

The answer is the self hosted integration runtime. You install it on a machine inside that network. It connects outward to the service, receives the work and performs it locally. The pipeline is defined in the cloud but the data movement happens where the data is.

The Azure integration runtime handles work between cloud systems, and a third type exists for running packages migrated from SQL Server Integration Services.

Copying and transforming

The copy activity handles movement between a source and a sink using the connectors the service provides.

Transformation is a separate concept. Mapping data flows are designed visually, with steps such as joins, aggregates, lookups and derived columns, and they run on Spark compute that the service manages. You do not create or size that cluster yourself.

A pipeline can also simply call something that already exists, such as a notebook or a stored procedure, when the transformation logic lives elsewhere.

Monitoring runs

Every pipeline run is recorded, with the status of each activity inside it. This is where most operational time is spent, and the monitoring section of the documentation covers reruns, alerts and the metrics available.

Who uses it

Azure Data Factory is used by data engineers and integration teams working on Azure, particularly where sources are split between cloud services and systems still running on premises.

Points to consider

The service is designed for Azure, so it is at its most straightforward within that environment.

Cost also depends on more than the number of pipelines. Activity runs, data movement and the compute used by data flows are billed separately, and the pricing page on the official site sets out how each is measured.

Getting started

The documentation includes quickstarts that create a data factory and build a first copy pipeline through the studio interface, along with tutorials covering data flows, triggers and the self hosted integration runtime.