Apache NiFi
A platform for automating the flow of data between systems, built and controlled from a visual interface.
Apache NiFi moves data between systems. Flows are built and changed in a browser by connecting processors together, and every piece of data carries its own attributes as it travels. The documentation describes NiFi as built to automate the flow of data between systems, with guaranteed delivery, buffering and a full record of where each item came from.
What is Apache NiFi?
Apache NiFi is a platform for automated and managed information flow between systems. It runs inside a Java virtual machine on a host operating system.
Rather than writing a program for each connection, you build a flow on a canvas in the browser. You drag in processors, connect them together and the flow starts moving data. Changes take effect while the flow is running.
The purpose of Apache NiFi
Enterprises rarely have systems that were designed to work together. The documentation describes the problem as connecting a massively distributed system of components that are loosely or not at all designed to work together, across service oriented architecture, APIs, IoT and big data.
NiFi is built to keep working when that environment misbehaves. Its documentation lists handling systems that fail, data arriving faster than it can be consumed and records that are oversized, undersized, corrupt or in the wrong format.
How a NiFi dataflow works
Five terms cover most of what happens.
- A FlowFile is one item moving through the system. It holds key and value attributes plus the content itself.
- A processor does the work. It routes, transforms or mediates FlowFiles.
- A connection joins two processors and acts as a queue that can be prioritised.
- The flow controller schedules processors and allocates threads to them.
- A process group bundles processors and connections so they can send and receive data through ports.
Three repositories sit behind all of this. The FlowFile repository is a write ahead log holding the state of active FlowFiles. The content repository stores the actual bytes. The provenance repository stores indexed events describing what happened to every FlowFile.
Who uses Apache NiFi
NiFi suits organisations with many connected systems and a need for delivery guarantees at scale. The documentation points to multi tenant environments that need fine grained access control, and to edge deployments through the MiNiFi child project.
Because flows are built visually, the people running a flow do not have to be the people who wrote the connector behind it.
Known constraints
NiFi is bound by the machine it runs on. The documentation gives a conservative estimate of roughly 50 MB per second read and write rate on modest disks, and identifies the FlowFile repository and the provenance repository as the points that limit throughput at scale.
Other limits it lists are the size of the Java heap and the efficiency of garbage collection, the RAM available for reading content and the number of threads the host can provide.
Starting with Apache NiFi
NiFi is controlled through an HTTP based command and control API served by its web server, and the web interface can be reached from any node in a cluster. A change made on one node replicates to the others automatically.
Clustering uses a zero leader design in which ZooKeeper elects a cluster coordinator and a primary node. Scaling up is done by adding nodes or by raising the number of concurrent tasks, and MiNiFi covers the smaller edge devices.
Key features of Apache NiFi
Capabilities described in the official documentation.
Guaranteed delivery
A persistent write ahead log records the state of every item, so delivery survives a restart.
Buffering and back pressure
Queues buffer data and apply back pressure when a downstream system cannot keep up.
Visual command and control
Flows are created and changed in the browser while data is moving through them.
Data provenance
Every event affecting an item is recorded, indexed and searchable, with replay available from a rolling buffer history.
Advantages of Apache NiFi
Factual advantages that follow from the features above.
Flows can be changed while running
Real time visual control means a flow is adjusted on the canvas rather than rebuilt and redeployed.
A traceable record of every item
Provenance data shows where a piece of data came from and what happened to it along the way.
Security controls in the platform
Two way SSL covers system to system and user to system connections, with pluggable authorisation and multi tenant access control.
Flows are assembled without code
Processors are configured through their properties, so a route between two systems is built rather than programmed.
Common use cases for Apache NiFi
Situations the official documentation describes this tool as being used for.
Connecting systems that use different protocols
NiFi bridges systems whose formats and protocols were never designed to match.
Absorbing data that arrives faster than it is consumed
Queues hold the excess and back pressure protects the system downstream.
Meeting audit and accountability requirements
Provenance records support compliance, security and accountability obligations.
Routing data based on its content
A flow inspects each item and sends it down a different path depending on what it contains.
Official website
Everything on this page is based on the official documentation for Apache NiFi. You can read the source here.
Apache NiFi official documentation