Apache Hadoop
A framework for storing and processing large datasets across a cluster of machines.
Apache Hadoop is a framework for distributed storage and processing. It splits files across many machines with HDFS, processes them with MapReduce and manages cluster resources with YARN. The documentation shows it running as a single Java process on one machine for testing, and notes that Hadoop has been demonstrated on clusters of 2000 nodes.
What is Apache Hadoop?
Apache Hadoop is a distributed computing framework. It provides both storage and processing across a group of machines that behave as one system.
Three modules do most of the work. HDFS, the Hadoop Distributed File System, holds the data. MapReduce processes it. YARN handles resource management and job scheduling.
What Apache Hadoop is used for
Hadoop exists for datasets and workloads that do not fit on one machine. Files are split across nodes, and the processing is sent to where the data already sits rather than pulling everything into one place.
The documentation demonstrates the model with example jobs run once the cluster is started, and notes that Hadoop has been demonstrated on clusters of 2000 nodes.
How the Hadoop modules fit together
HDFS stores the files. Its NameNode keeps the directory of where everything is, and publishes a web interface on port 9870.
YARN decides which work runs where and how much of the cluster it may use. Its ResourceManager has a web interface as well, on port 8088.
MapReduce is the processing model that runs on top of both, and the documentation uses it for the example jobs in the single node walkthrough.
Who works with Apache Hadoop
Hadoop is run by teams comfortable with Linux administration. The documentation states that GNU/Linux is supported as both a development and a production platform, and the setup involves editing shell and XML configuration files by hand.
It is also used by people learning distributed processing, because the same distribution runs as a single process on one machine.
Practical limitations
Java is required, and the supported versions are listed separately in the Hadoop Java Versions documentation, so the version you install has to be checked against that page.
SSH is required as well. The daemon management scripts need ssh installed and sshd running, and passphraseless SSH access has to be configured before the daemons will start. The documentation recommends pdsh for better SSH resource management.
Security is not covered by the basic setup. The documentation states that all production Hadoop clusters use Kerberos to authenticate callers, that Kerberos integration is not part of the single node instructions and that production deployments should connect to the organisation's own Kerberos infrastructure.
Setting up a single node cluster
The documented order is short.
- Download a stable release from the Apache download mirrors and unpack it.
- Set JAVA_HOME in etc/hadoop/hadoop-env.sh.
- Edit the XML configuration files, including core-site.xml and hdfs-site.xml.
- Set up passphraseless SSH access.
- Format the HDFS filesystem.
- Start the daemons with sbin/start-dfs.sh and, if you need YARN, sbin/start-yarn.sh.
- Create the HDFS directories the examples need and run an example job.
Three modes are available. Standalone runs everything in a single Java process and is useful for debugging. Pseudo distributed runs each daemon as its own process on one node. Fully distributed spreads the daemons across many nodes and is covered by the separate cluster setup documentation.
Key features of Apache Hadoop
Capabilities described in the official documentation.
Distributed file storage
HDFS splits files across the machines in a cluster and tracks their location through the NameNode.
MapReduce processing
Work is expressed as a MapReduce job and executed close to the data it reads.
Resource management with YARN
YARN schedules jobs and decides how much of the cluster each one is allowed to use.
Web interfaces for the cluster
The NameNode publishes an interface on port 9870 and the ResourceManager on port 8088.
Advantages of Apache Hadoop
Factual advantages that follow from the features above.
The same distribution runs at any size
One release runs standalone in a single Java process, pseudo distributed on one node or fully distributed across many.
Storage and processing scale together
Adding nodes adds both capacity and compute, and the documentation notes clusters demonstrated at 2000 nodes.
Cluster state is visible in a browser
The NameNode and ResourceManager interfaces report the condition of storage and of running work.
Blocks are replicated across nodes
Each block is written to more than one machine, so losing a node does not mean losing the data on it.
Common use cases for Apache Hadoop
Situations the official documentation describes this tool as being used for.
Storing datasets larger than one machine
HDFS spreads a file across the cluster and keeps track of where every part of it lives.
Running batch jobs across a cluster
MapReduce jobs are submitted through YARN and executed on the nodes that hold the data.
Learning distributed processing locally
Standalone mode runs the whole framework in a single Java process, which the documentation recommends for debugging.
Retaining data that is read rarely
Large volumes are kept on cluster storage where the cost suits data that is queried only occasionally.
Official website
Everything on this page is based on the official documentation for Apache Hadoop. You can read the source here.
Apache Hadoop official documentation