ONNX
An open format for representing trained models, so a model built in one framework can run in another.
ONNX is an open format for machine learning models. A model trained in one framework is exported into the format, which describes its structure and its learned values in a way that does not depend on the framework that produced it. Runtimes then load that file and run the model.
The coupling ONNX breaks
A model trained in a framework is normally tied to it. Serving that model means installing the framework in the serving environment.
That couples two decisions which have little to do with each other.
Training frameworks are chosen for how well they support developing a model: the research your work builds on, the tooling, what your team knows.
Serving environments are chosen for entirely different reasons: what the application is written in, what the target hardware is, how small the deployment has to be.
Requiring them to match means one constrains the other, usually badly.
What the format does
ONNX describes a model as a graph of operations plus the learned values.
The operations are specified in the format itself rather than referring to any framework's implementation. That is what makes the description portable: a runtime reads the file and executes the operations, without knowing or caring what produced it.
Major training frameworks can export into this format, and runtimes exist across platforms and languages.
What that enables
Serving without the training framework. A runtime is far lighter than a framework built for training. For a deployment where size matters, that difference is substantial.
Running where the framework cannot. Runtimes exist for devices and browsers. A model can run close to the user, without a network call and without the input data leaving the device.
Serving in another language. A model trained in Python is loaded by a runtime in the language the serving application uses, so the application is not forced into Python for the sake of the model.
Where exports go wrong
This deserves stating plainly, because it is the practical difficulty.
Not every model exports cleanly. A model using an operation the format does not cover, or relying on a framework specific construct, may fail to export or may export into something that behaves differently.
The failure mode that hurts is discovering this at deployment, after the model is finished and a date has been set.
The sensible approach is to attempt the export early, while the model is still being developed, and confirm that the exported version produces the same outputs as the original. That turns a potential crisis into a known constraint you can design around.
The operator set
The format specifies which operations a model may use, with versions.
That specification is what runtimes implement, and it defines the boundary of what can be represented. Checking that the operations your model needs are covered is part of planning, particularly for anything unusual.
Who uses ONNX
ONNX is used by teams putting models into production, particularly where the serving environment differs from the training one: applications in other languages, deployments on devices, or serving where a lightweight runtime matters.
Points to consider
It is an interchange format, not a training framework. Models are trained elsewhere and exported.
Export coverage varies by framework and by model, and testing the exported model against the original is necessary rather than optional.
Runtime support for operations also varies, so a model that exports successfully should still be verified on the runtime that will serve it.
Getting started
The documentation covers the format, the operator set and the export path from each supported framework. Exporting an existing model and comparing its outputs against the original is the first thing worth doing, and it establishes whether the path is viable before anything depends on it.
Key features of ONNX
Capabilities described in the official documentation.
A framework independent representation
The format describes a model's operations and values without depending on the framework that trained it.
Export from several frameworks
Major training frameworks can export into the format, so a model is not locked to the one that produced it.
Runtimes on many platforms
Runtimes load exported models on servers, devices and in browsers, across several languages.
A defined operator set
The operations a model may use are specified with versions, so what a file contains is documented.
Advantages of ONNX
Factual advantages that follow from the features above.
Training and serving can differ
The environment a model runs in does not have to match the one it was trained in, which decouples two decisions.
Serving does not need the framework
A runtime is generally lighter than a full training framework, which matters where the model has to be deployed.
Models can run where the framework cannot
Runtimes for devices and browsers put a model in places a training framework would not run.
The file documents the model
Because operations are specified, an exported model can be inspected rather than treated as opaque.
Common use cases for ONNX
Situations the official documentation describes this tool as being used for.
Deploying without the training framework
An exported model is served by a lightweight runtime rather than by installing the framework that trained it.
Running a model on a device
A model is exported and run by a runtime on hardware where the training framework would not be practical.
Serving from a different language
A model trained in Python is loaded by a runtime in another language used by the serving application.
Keeping deployment independent of training
A framework change during development does not require the serving path to be rebuilt.
Official website
Everything on this page is based on the official documentation for ONNX. You can read the source here.
ONNX official documentation