Bigeye
A data quality monitoring platform that learns normal behaviour for a table and alerts when readings depart from it.
Bigeye monitors data quality. It applies metrics to tables, such as row counts, null rates and value distributions, tracks those metrics over time, and raises alerts when a reading departs from what the history suggests is normal. Thresholds can be set explicitly or derived from the observed behaviour of each metric.
The problem with writing rules
The traditional approach to data quality is to write rules. This column must not be null. This value must fall between one and one hundred. This key must be unique.
The approach is sound and it has a practical difficulty that limits it.
Somebody has to write the rules, for every column that matters, across every table that matters. In a warehouse with hundreds of tables and thousands of columns, that is a great deal of work before any monitoring exists.
Worse, most of the interesting problems are not rule violations.
A table receives five million rows a day. One day it receives two million. No rule is broken. Every value is valid. The data is simply incomplete, and no reasonable set of rules would have caught it, because there is nothing invalid about two million rows.
Measuring, then comparing against history
Bigeye approaches this by measuring rather than by asserting.
Metrics are applied to tables: row counts, null rates, distinct counts, distributions of values, freshness. These are measured repeatedly, and the readings accumulate into a history.
Once there is history, the question changes. Instead of asking whether a value is valid, it asks whether today's reading is consistent with how this metric has behaved before.
The table that normally receives five million rows and today received two million is now detectable, because the history says what normal looks like.
This is a different question from rule checking and it catches a different class of problem, which is why the two approaches complement each other rather than competing.
Where thresholds come from
Expectations can be derived from observed behaviour or stated explicitly, and both have their place.
Derived suits metrics where normal is a range. Row counts vary. Weekdays differ from weekends, months have different lengths, business volumes fluctuate. Choosing a fixed number that catches real problems without firing on ordinary variation is difficult, and people usually set it too loose to avoid noise, at which point it catches nothing.
Explicit suits requirements that are definite. A primary key must be unique. A percentage must lie between zero and one hundred. There is no need to observe history to know this, and stating it directly is clearer.
The practical value of derived thresholds is that monitoring can begin without a specification exercise. Point it at tables, let it observe, and coverage exists without anyone having written a rule per column.
The corresponding limitation is that it needs time to observe. A new table has no history, so there is nothing yet to compare against.
There is a second caveat worth stating. If a table has been quietly wrong for months, that becomes the observed normal. Learned expectations describe what has been happening, which is not necessarily what should be happening.
What the history is worth beyond alerting
The accumulated measurements are useful independently of alerts.
When a problem is found, the first question is when it started. A recorded series answers that directly: the metric behaved one way until a particular date and differently afterwards.
That narrows investigation considerably. Whatever changed, changed then, and the search is limited to deployments, source changes and configuration changes around that date.
Without history, establishing when something began is guesswork, and it is usually the most time consuming part of a data investigation.
Who uses it
Bigeye is used by data engineering and analytics teams responsible for tables that feed reporting and downstream systems. It is most relevant where the number of tables has grown past what rules written by hand can cover, and where problems are currently found by people noticing that a report looks wrong.
Points to consider
Bigeye is a commercial platform. The official documentation is the reference for supported data sources, deployment and available metrics.
Monitoring needs observation before it is effective. Expectations derived from history require history, so there is a period after connecting a source during which coverage is building rather than complete.
Learned behaviour reflects what has occurred. A long standing problem may be absorbed as normal, which is an argument for combining derived thresholds with explicit rules where a requirement is actually known.
Alerts require ownership. Detection is only valuable if a person receives the alert and acts, and monitoring that reports to nobody in particular produces a record of problems rather than fixes.
Getting started
The official documentation covers connecting sources, configuring metrics and how thresholds are set. Applying metrics to a handful of tables that feed an important report, and letting them observe for a period, is a practical way to see what the monitoring surfaces before extending coverage.
Key features of Bigeye
Capabilities described in the official documentation.
Metrics tracked over time
Measurements such as row counts and null rates are recorded repeatedly so history accumulates.
Thresholds derived from history
Expected ranges are inferred from how a metric has behaved rather than requiring a number to be chosen.
Explicit rules where they are known
Where a requirement is definite, a fixed threshold can be stated instead of an inferred one.
Alerting on departures
A reading outside the expected range raises an alert to the people responsible for the table.
Advantages of Bigeye
Factual advantages that follow from the features above.
Monitoring starts without defining every rule
Deriving expectations from history avoids the requirement to specify thresholds for every column first.
Normal variation is accounted for
Because expectations come from observed behaviour, ordinary fluctuation does not raise alerts.
Problems surface before reports do
Checking data directly finds issues that a completed pipeline run would not reveal.
History shows when something changed
A recorded series of measurements indicates when behaviour shifted, which narrows investigation.
Common use cases for Bigeye
Situations the official documentation describes this tool as being used for.
Watching tables that feed reporting
Metrics on the tables behind published figures catch problems before the figures are seen.
Detecting a change in an upstream feed
A shift in distribution or volume indicates that a source system began behaving differently.
Monitoring completeness of loaded data
Null rates and row counts reveal partial loads that finished without raising an error.
Establishing when a problem began
The recorded history of a metric shows the point at which behaviour departed from normal.
Official website
Everything on this page is based on the official documentation for Bigeye. You can read the source here.
Bigeye official documentation