Apache Flume

The problem Flume addresses

An application writes logs to a file on the machine it runs on. That is fine with one machine. With two hundred machines, the logs are in two hundred places and nothing can be analysed across them.

Something has to collect from every machine and deliver to one place, keep working when the destination is slow and not lose data when a machine restarts.

Apache Flume is a service built for that job. The project describes it as a distributed service for collecting, aggregating and moving large amounts of log data.

The agent, and its three parts

Everything in Flume is built from one structure. An agent is a process containing:

A source. Where data comes in. It might read a directory of files, follow a file as it is written, or listen on a network port.

A channel. Where data sits between arriving and leaving. This is a buffer, and its type is one of the more consequential configuration choices.

A sink. Where data goes. Cluster storage, another agent or another system.

Data moves source to channel to sink, and an agent is defined in a properties file rather than written as code.

Why the channel matters

The channel is not just a buffer. It is where Flume's delivery behaviour is decided.

A memory channel holds events in memory. It is fast, and everything in it is lost if the agent stops.

A file channel writes events to disk. It is slower, and events survive a restart.

The choice is a straightforward trade between throughput and durability, and it should follow how much the data matters rather than which number looks better in a test.

How events are handed on

Flume moves data transactionally between stages. An event is not removed from a channel until the next stage has confirmed it took it.

The practical effect is that a failure between stages results in the event being sent again rather than disappearing. Duplicates are possible; silent loss is much less likely.

Chaining agents

Agents can deliver to other agents, and this is how Flume is usually deployed at any scale.

Collecting agents run on each application machine. They deliver to a smaller number of aggregating agents. Those aggregating agents write to the final destination.

The destination therefore receives connections from a handful of agents rather than from every machine in the estate, and files written to storage are fewer and larger.

Interceptors

An interceptor sits in the path of events inside an agent and can inspect or modify them. Common uses are adding a timestamp, adding the originating host name, or filtering out events that do not match a pattern.

They are configured alongside the source and are intended for small adjustments rather than substantial processing.

Who uses Flume

Flume is used by operations and platform teams collecting log and event data into a central store, particularly in estates built around Hadoop. It is configured rather than programmed, so it suits teams who want a collection pipeline without an application to maintain.

Points to consider

Flume is oriented towards collection and delivery. It is not a general purpose integration tool, and reshaping data is expected to happen after it lands.

Its configuration is also file based, which makes a large deployment a set of files to manage carefully across many machines.

As with any long established project, checking the official site for current release activity and project status is worth doing before choosing it for new work.

Getting started

The project site publishes a user guide covering agent configuration, every source, channel and sink type, interceptors and multiple agent arrangements. A first exercise is usually an agent that follows a local file and writes what it reads to another file.