Azure Data Explorer

A different kind of question

Analytical databases are generally built for structured questions. Revenue by region by quarter, joined across a properly modelled set of tables.

Log and telemetry data produces a different kind of question, and the difference is in how the questions arrive.

Something is wrong. Errors increased at some point in the last hour. Which service. Which version. Which customers. Did anything deploy around then. Is it one region or all of them.

Each question follows from the answer to the last. Nobody knows in advance which query they will need, because the third query depends on what the second showed.

That pattern of work has requirements. Queries must return quickly enough that a line of enquiry can be followed. And the language must make it easy to take a result and narrow it further.

A query as a sequence

The Kusto Query Language reflects this directly. A query is a sequence of operations, with records flowing from each step to the next.

Start with a table. Filter to a time range. Filter to a service. Summarise by error type. Sort. Take the top ten.

Each step receives what the previous produced.

The contrast with SQL is instructive. SQL describes the result in one statement, with clauses in a fixed order that does not match the order the work is conceived in. That is a good fit for structured queries against a model.

For exploration, a pipeline matches how the work actually proceeds. Add a line to the end and the query narrows further. Remove a line and it broadens. The text of the query grows in the same direction as the investigation.

This is a genuine ergonomic difference for this kind of work, and it is why the language exists rather than the service simply accepting SQL.

Speed as a requirement rather than a nicety

For a scheduled report, a query taking two minutes is fine. Nobody is watching.

For an investigation, two minutes per query is the difference between following a line of enquiry and abandoning it. A person will ask fifteen questions in ten minutes if answers are quick, and three if each involves waiting.

Data is indexed on ingestion so that filtering across very large volumes returns quickly. That is what makes the exploratory pattern viable rather than merely possible.

Data that arrives and stays

The service is built for a particular shape of data.

Records arrive continuously from applications, devices and systems. Each carries a timestamp. They are not edited afterwards, because a log entry describes something that happened and what happened does not change.

Continuous ingestion means recent data becomes queryable shortly after it arrives, which matters when the question is about what is happening now rather than what happened last month.

Data that is updated in place, or that requires extensive joining across entities, does not fit this shape and is better served by a warehouse or a relational database.

Time series work

Because the data carries timestamps and arrives continuously, functions for working with measurements over time are part of the language.

Aggregating into intervals, comparing periods, and examining how values move are all directly expressible rather than requiring elaborate constructions.

This is what makes the service useful for telemetry as well as for logs. Equipment readings and application metrics are the same shape of data as log records, and the same tools apply.

Who uses it

Azure Data Explorer is used by engineering teams investigating application behaviour, operations teams monitoring systems, security teams examining events, and analysts working with telemetry. Its users are more often diagnosing than reporting, which reflects what it is built for.

Points to consider

This is an Azure service and the official documentation is the reference for capabilities, ingestion methods and pricing.

The query language has to be learned. It is not difficult and it is not SQL, so a team accustomed to SQL should expect a short adjustment.

It is purpose built rather than general purpose. Using it for workloads that suit a warehouse means fighting the design, and the reverse is equally true.

Retention and cost are connected, as with any system ingesting continuously. Deciding how long data is kept, and at what detail, is a decision to make deliberately rather than discovering through the bill.

Getting started

The official documentation covers ingestion, the query language and cluster configuration. Working through the query language material with a sample dataset is the fastest route, because the pipeline model becomes intuitive after a few queries and the rest follows from it.