Airbyte

Airbyte explained

Airbyte is a platform for moving data from where it is produced to where it will be analysed. The documentation describes it as an open source data movement platform, and open source is the part that shapes most of the decisions around it.

Because the connectors are published openly, you can read them. When a sync brings back a field in an unexpected shape, the answer is in code you can look at rather than behind a support ticket.

Four things to understand

Source. Where data comes from. A database, an application, a file store or an API.

Destination. Where data goes. Usually a warehouse, a lake or another database.

Stream. One table or object within a source, such as a single table in a database or one endpoint in an API.

Connection. The link between a source and a destination. It holds the schedule, the streams selected and the sync mode for each of them.

How sync modes differ

The sync mode decides what happens on every run after the first, and it is the setting most worth getting right.

• Full refresh, overwrite. Everything is read again and replaces what was there. Simple, and fine for small data sets.
• Full refresh, append. Everything is read again and added, leaving a history of every run.
• Incremental, append. Only new or changed records are read and added to the destination.
• Incremental, append and deduplicate. Only new or changed records are read, and a primary key is used so the destination keeps one current row per record.

Adding a connector that does not exist

Every catalogue has gaps, and internal systems are never in one. Airbyte documents two routes.

The Connector Builder is a low code interface where you describe an API: its base URL, how it authenticates, which endpoints to call and how it paginates. Much of what an API connector needs can be expressed there without writing code.

The connector development kit covers the rest, for sources whose behaviour is more unusual than a form can capture.

Running it yourself

Airbyte can be deployed on your own infrastructure. The documentation covers a local install through its command line tool and deployment on Kubernetes with Helm for a longer lived environment.

This is the reason many teams choose it. When source data is sensitive, a pipeline that never leaves your own network removes a set of questions that would otherwise need answering.

Who it suits

Airbyte fits data teams that want control over their pipelines, teams with sources that are not in any managed catalogue and organisations whose data cannot be sent through a third party service. Running the self hosted version does require somebody comfortable operating the deployment.

What to bear in mind

Connectors in an open catalogue vary in maturity. The documentation marks connector support levels, and that marking is worth checking for any source you intend to depend on.

Self hosting is also real work. The platform has to be deployed, upgraded, monitored and resourced, and that responsibility sits with you rather than with a vendor.

Getting started

The documentation includes a quickstart that runs Airbyte locally, then walks through adding a source, adding a destination and creating the connection between them. There are separate sections for each connector, for deployment options and for the Connector Builder.