Amazon EMR
An AWS platform for running open source big data frameworks such as Spark, Hive and Presto on managed clusters.
Amazon EMR is a managed platform for running open source data processing frameworks on AWS. It handles provisioning the machines, installing the frameworks and configuring them so a cluster is ready to accept work. EMR can run on virtual machines, on Kubernetes or without a cluster at all through its serverless option.
Amazon EMR in outline
Amazon EMR is a platform for running open source big data frameworks on AWS. The frameworks are the familiar ones: Apache Spark, Apache Hive, Presto, Trino, HBase, Flink and others. What EMR provides is everything around them.
Setting these frameworks up by hand is a substantial job. Machines have to be provisioned, software installed on each, versions matched so components work together and configuration applied consistently. EMR does that when you create a cluster.
Releases fix the versions
An EMR release is a defined set of framework versions packaged and tested together.
This matters more than it first appears. Big data frameworks depend on each other, and mismatched versions cause failures that are difficult to diagnose. Choosing a release means choosing a combination that is known to work, rather than assembling one.
Three ways to run it
On virtual machines. The original model. A cluster of instances is created, the frameworks are installed on them and jobs are submitted to the cluster.
On Kubernetes. Jobs run on an existing Kubernetes cluster, which suits organisations that already operate one and want data processing alongside their other workloads.
Serverless. No cluster is defined. An application is submitted and AWS provisions what it needs for the run.
The management guide covers each as a separate deployment option, with its own setup and operational guidance.
The three node roles
A cluster running on virtual machines has:
- One primary node, which coordinates work across the cluster.
- Core nodes, which run tasks and also store data on the cluster file system.
- Task nodes, which run tasks but store nothing.
The difference between core and task nodes is worth remembering when scaling. Removing a task node takes away processing capacity. Removing a core node also takes away a copy of stored data.
Storage that outlives the cluster
Clusters read from and write to object storage directly. This is what makes a temporary cluster practical: data is not trapped on machines that are about to be terminated.
The pattern that follows is common. Create a cluster, run the work against data in storage, write the results back to storage, terminate the cluster. Nothing of value is lost when it goes.
Who EMR is for
EMR suits teams with substantial processing written for open source frameworks, and organisations moving an existing Hadoop or Spark estate onto AWS. It is aimed at data engineers, since jobs are written and submitted rather than configured in a form.
Points to weigh
EMR manages the platform, not your jobs. Spark tuning, partitioning and memory settings remain your responsibility, and a poorly written job runs no better here than anywhere else.
There are also several deployment options, each documented separately. Choosing between them is an early decision, because the operational model differs considerably between a cluster you run and a serverless application you submit.
Getting started
The management guide includes a tutorial that launches a cluster, runs a sample Spark application against data in object storage and then terminates the cluster. There are separate guides for the Kubernetes and serverless deployment options.
Key features of Amazon EMR
Capabilities described in the official documentation.
Framework versions installed for you
An EMR release bundles versions of Spark, Hive, Presto and other components that have been tested together.
Several deployment options
Work can run on virtual machine clusters, on a Kubernetes service or through a serverless option with no cluster to define.
Three node roles
A primary node coordinates the cluster, core nodes run tasks and hold data, and task nodes add processing capacity only.
Reads directly from object storage
Clusters read and write data in object storage, so data does not have to be loaded onto the cluster before processing.
Advantages of Amazon EMR
Factual advantages that follow from the features above.
No manual framework installation
Selecting a release gives a working set of frameworks, rather than installing and matching versions across machines.
Clusters sized for the job
Because a cluster is created on request, capacity can match the work rather than being fixed by what was purchased.
Data survives the cluster
Keeping data in object storage means a cluster can be terminated once its work is done without any data being lost.
Capacity can change while work runs
Nodes can be added to or removed from a running cluster, so a long job is not fixed to the size it started at.
Common use cases for Amazon EMR
Situations the official documentation describes this tool as being used for.
Large scale Spark processing
A cluster runs Spark jobs over data held in object storage and is terminated once the processing finishes.
Interactive SQL over a data lake
A query engine installed on the cluster answers analytical SQL against tables stored in the lake.
Migrating an existing Hadoop estate
Jobs from an on premises Hadoop platform are moved onto managed clusters running the same open source frameworks.
Running processing beside container workloads
Jobs run on a Kubernetes cluster the organisation already operates rather than on a separate set of machines.
Official website
Everything on this page is based on the official documentation for Amazon EMR. You can read the source here.
Amazon EMR official documentation