Apache Druid
A database built for fast analytical queries over event data, where time is treated as a first class column.
Apache Druid is an analytics database designed for queries that slice and filter large volumes of event data. It stores data in columns, partitions it by time and builds indexes that let a query skip most of the data before reading any of it. Druid can take data from a stream as it arrives as well as loading it in batches.
The kind of question Druid is built for
Apache Druid is an analytics database. The documentation describes it as designed for fast analytics on large data sets, and the queries it targets have a recognisable shape.
They filter. They group. They cover a time range. Something like: over the past six hours, how many events occurred, broken down by region and device type. Not: join this table to four others and produce a reconciliation.
Understanding that shape explains most of Druid's design.
Time is not an ordinary column
In most databases, a timestamp is a column like any other. In Druid it decides how data is physically arranged.
Data is split into segments by time interval. A segment holds the rows for its interval, stored column by column with indexes built over the values.
The effect on a query covering the last six hours is direct. Segments outside that range are not considered at all. Within the segments that remain, indexes narrow the work further, and only the columns the query mentions are read.
Getting data in
Druid supports two routes, and many deployments use both.
Streaming ingestion reads continuously from a message stream. Records become queryable shortly after they arrive, which is what makes Druid suitable for dashboards showing current activity.
Batch ingestion loads data from files. This is used for backfilling history and for reprocessing a period after a correction.
Rollup, and what it costs
Druid can aggregate as it ingests. If rollup is enabled, rows that share the same dimension values within a time bucket are combined, and measures such as counts and sums are stored instead of the individual rows.
Storage drops considerably. So does the ability to answer questions about individual events, because those rows no longer exist. It is a decision made at ingestion time and it is not reversible for data already stored that way.
How a deployment is arranged
Druid runs several process types with different responsibilities. The documentation groups them into three server roles:
- Master servers manage the cluster, deciding where segments live and coordinating ingestion tasks.
- Query servers receive queries, work out which data nodes hold the segments needed and combine the results.
- Data servers hold segments and perform the actual scanning and aggregation.
Alongside these sits deep storage, which holds the authoritative copy of every segment. Losing a data node does not mean losing data, because the segment can be loaded again from deep storage.
Who uses Druid
Druid suits teams serving interactive analytics over event data to many users, particularly where the data arrives continuously and recent activity matters. It is operated by data platform engineers, while analysts and applications query it through SQL.
Where it is not the right fit
Druid is not a general purpose database. Large joins are not its strength, and the documentation is clear about the workloads it targets.
It is also not a transactional system. Data is ingested and queried, not updated record by record the way an application database would.
Operationally it has several process types to run, which is more involved than deploying a single server database.
Getting started
The documentation includes a local quickstart that runs Druid on one machine, loads a sample data set and runs queries against it from the built in console. Separate sections cover ingestion methods, segment design, rollup and cluster deployment.
Key features of Apache Druid
Capabilities described in the official documentation.
Time based partitioning
Data is split into segments by time, so a query covering one day never has to consider segments from other days.
Columnar storage with indexes
Values are stored by column and indexed, so filtering and aggregating read only the columns a query mentions.
Streaming and batch ingestion
Druid can consume from a message stream continuously and also load data from files in a batch job.
Optional rollup at ingestion
Rows sharing the same dimensions within a time bucket can be combined as data is ingested, reducing what is stored.
Advantages of Apache Druid
Factual advantages that follow from the features above.
Recent data is queryable quickly
Streaming ingestion makes records available to queries shortly after they arrive rather than after a scheduled load.
Filters narrow the work sharply
Time partitioning and indexes mean a filtered query examines a small fraction of the stored data.
Data survives node loss
Segments are written to deep storage as well as being served from data nodes, so the copy of record sits outside the cluster.
Storage shrinks where detail is not needed
Rollup combines rows sharing dimension values at ingestion, so a table holds aggregates rather than every event.
Common use cases for Apache Druid
Situations the official documentation describes this tool as being used for.
Operational dashboards over events
A dashboard filters and groups recent event data, with each interaction answered by a fresh query.
Product and clickstream analysis
Analysts break down user events by attributes and time windows to see how behaviour differs across segments.
Monitoring metrics at volume
Metrics arriving continuously from many sources are ingested from a stream and queried by time range.
Powering a customer facing analytics view
An application queries Druid directly so its own users can filter and group event data themselves.
Official website
Everything on this page is based on the official documentation for Apache Druid. You can read the source here.
Apache Druid official documentation