Apache Spark

When one machine is not enough

Processing data on a single machine works until the data outgrows it.

The limits arrive in two ways. The data no longer fits in memory, so processing either fails or becomes extremely slow as it moves to and from disk. Or the processing takes longer than the time available, so a job that must finish overnight is still running in the morning.

Spark addresses both by dividing the work across many machines. Each holds part of the data and processes it at the same time as the others.

The property that matters most is that the code does not change with the scale. The same job that processes a small dataset processes a much larger one on more machines.

Recording work rather than doing it

Spark's execution model surprises people coming from ordinary programming, and understanding it explains a great deal about how it behaves.

Operations are not executed as they are written. They are recorded.

Filtering a dataset, then selecting columns, then grouping does nothing. Spark builds a description of the requested work. Only when a result is actually needed does anything run.

This is deliberate, and it is what allows the engine to plan.

Because the whole sequence is known before execution, the engine can rearrange it. If a filter later removes most rows, applying it earlier avoids processing data that will be discarded. If only three columns are used, the others need not be read at all.

None of this is possible if each step runs as it is written. The planning depends on seeing the whole job first.

The practical consequence for anyone new to Spark is that a job appears to do nothing for a long time and then does everything at once. That is the model working as intended.

Shuffles

If one concept explains Spark performance, it is the shuffle.

Some operations work on data where it already sits. Filtering rows or transforming a column happens on each machine independently, with no communication.

Other operations do not. Grouping by a key requires all rows with that key to be on the same machine. Joining two datasets requires matching rows brought together.

Moving data between machines to satisfy this is a shuffle, and it involves the network, which is far slower than memory or local disk.

Slow Spark jobs are very frequently explained by shuffles: too many, or moving too much data. Tuning consists largely of reducing them or reducing what they carry.

This also explains why the same logic expressed two ways can differ enormously in speed. The difference is whether the arrangement forces data to move.

Several languages, one engine

Spark provides interfaces in Python, Scala, Java, SQL and R.

The practical value is that a team need not standardise on one language to use the engine. Data engineers working in Python and analysts writing SQL both contribute, and both go through the same engine.

Whichever interface is used, the same planning applies underneath, so the choice is about who is writing rather than about what the engine can do.

Batch and streaming

Data arriving continuously is processed through interfaces that mirror those for batch work.

The benefit is not having two separate bodies of logic. Where a transformation must apply both to historical data and to records arriving now, expressing it once rather than twice avoids the two drifting apart, which is a persistent problem where they are implemented separately.

Who uses it

Spark is used by data engineers and analytics teams processing volumes beyond what one machine handles. It appears in pipeline work, in preparing data for warehouses and for machine learning, and it runs on managed services as well as on clusters teams operate themselves.

Points to consider

Spark is an Apache project under an open source licence, and the official documentation is the reference for its interfaces and configuration.

It is not appropriate for small data. Distribution has overhead, and a dataset one machine handles comfortably is processed faster by a single process. Reaching for Spark by default produces slower jobs and more complexity than the work requires.

Operating a cluster is real work. Managed services remove much of it, which is why a large proportion of Spark usage now happens on them.

Performance requires understanding shuffles. A team using Spark without that understanding will write jobs that work and run far more slowly than necessary.

Getting started

The official documentation covers the interfaces, the execution model and configuration. Running a job that groups a dataset and examining the execution plan makes shuffles visible, which is the single most useful thing to understand before writing anything substantial.