CatBoost
A gradient boosting library built around handling categorical columns without the leakage that naive encoding causes.
CatBoost is an open source gradient boosting library. Its distinguishing focus is categorical data: rather than requiring category columns to be converted beforehand, it encodes them using statistics computed in a specific ordered way that avoids a subtle form of information leakage. It also uses a symmetric tree structure and aims for usable results from default settings.
The problem CatBoost is built around
Most business data is full of categories. Product codes, regions, channels, customer segments, device types.
Tree based models need numbers, so categories have to be turned into numbers somehow. How that is done turns out to matter a great deal, and one popular approach fails in a way that is genuinely difficult to spot.
Why naive category encoding fails
A sensible sounding approach is to replace each category with the average outcome observed for it. If customers in one region convert at twelve per cent, that region becomes 0.12. The model gets a useful numeric signal.
The problem is that the record you are encoding is included in the average you encode it with. Its own outcome contributed to the number it is now being given as an input.
The model is therefore being shown a value that partly contains the answer. Training performance looks excellent. Performance on new data, where no such hint exists, is much worse.
This is called leakage, and it is one of the more common ways a machine learning project produces a model that looked good and then failed.
The ordered approach
CatBoost addresses this directly. The statistic used to encode a record is computed only from records that precede it in an ordering. A record never contributes to its own encoding.
The result is an encoding that carries the useful signal without the hint. The documentation sets out the mechanism in detail, and it is applied by the library rather than being something to implement.
Because this is handled internally, category columns are simply declared as categorical. There is no encoding step to write, and no risk of applying it differently at training and prediction time.
Symmetric trees
The second distinguishing choice is structural. Every node at the same level of a tree uses the same split condition, rather than each branch choosing independently.
That is less flexible, and it makes the trees compact and prediction fast. Evaluating a symmetric tree is a very simple operation, which matters when predictions are served at volume rather than computed in a notebook.
Defaults that work
Boosted tree libraries typically require tuning before performing well, and the search over settings takes time and compute.
CatBoost aims to produce reasonable results from its defaults. That does not remove the value of tuning, and it does mean a first useful model arrives quickly, which is exactly what is wanted when the question is whether the problem is solvable at all.
Who uses CatBoost
CatBoost is used where data is heavily categorical, which covers a great deal of retail, marketing and operational data. It is also chosen where prediction speed matters, and by teams who would rather not maintain their own encoding logic.
Points to consider
The categorical handling is the reason to choose it. For data that is entirely numeric, that advantage does not apply and other implementations may train faster.
Symmetric trees trade flexibility for compactness, and on some problems a less constrained structure performs better. Comparing implementations on your own data remains worthwhile.
As with any boosted tree library, this is for structured data rather than images, audio or raw text.
Getting started
The documentation covers installation, declaring categorical features, training a first model and the parameters that most affect results, along with the model analysis tools. Training once with categories declared and once with them encoded by hand demonstrates what the library is doing for you.
Key features of CatBoost
Capabilities described in the official documentation.
Categorical handling built in
Category columns are declared and encoded by the library rather than being converted by hand beforehand.
Ordered encoding
Statistics used to encode a category are computed from earlier records only, which prevents a record informing its own encoding.
Symmetric trees
Each level of a tree uses the same split condition across the level, which keeps trees compact and prediction fast.
Model analysis tools
Feature contribution and interaction reporting are provided so what the model relies on can be examined.
Advantages of CatBoost
Factual advantages that follow from the features above.
A common source of error is removed
Encoding categories with ordered statistics prevents the leakage that naive target based encoding introduces.
Less preparation before training
Because category handling is internal, a substantial preparation step does not have to be written and maintained.
Defaults produce usable results
The library aims to perform reasonably without tuning, which makes a first useful model quick to obtain.
Prediction is fast
The symmetric tree structure keeps models compact, which matters when predictions are served at volume.
Common use cases for CatBoost
Situations the official documentation describes this tool as being used for.
Data dominated by categorical columns
A model is trained where most inputs are categories, such as product, region and channel identifiers.
Avoiding a hand written encoding step
Category encoding is left to the library instead of being implemented and kept consistent across training and prediction.
Getting a reasonable model quickly
Default settings produce a usable model early, which establishes what performance is achievable.
Serving predictions at volume
Compact models keep prediction time low where a service answers many requests.
Official website
Everything on this page is based on the official documentation for CatBoost. You can read the source here.
CatBoost official documentation