Apache Hive
A data warehouse system that lets you query very large files in distributed storage using SQL.
Apache Hive is a distributed data warehouse system built on Apache Hadoop. It lets you read, write and manage large volumes of data held in distributed storage by writing SQL instead of writing distributed processing code. Hive also stores the table definitions for that data in a metastore, which other query engines can read.
What is Apache Hive?
Apache Hive is a data warehouse system that sits on top of Apache Hadoop. Its purpose is to let you work with very large amounts of data using SQL, even though that data is stored as ordinary files spread across many machines.
The project describes Hive as a distributed and fault tolerant system that makes it possible to read, write and manage petabytes of data in distributed storage. The important part is the interface. You write a select statement and Hive works out how to run it across the cluster.
The problem Hive was built to solve
Before Hive, getting an answer out of a Hadoop cluster meant writing a distributed processing job by hand. That is slow work and it requires a programmer for every question.
Hive changed the interface. A table is defined once, pointing at a folder of files and describing the columns inside them. After that, anyone who can write SQL can query the data. Hive turns the statement into a distributed job and returns the result.
How a query runs in Hive
- You create a table definition that names the columns, their types and the location of the files.
- Hive records that definition in the Hive Metastore.
- You submit a HiveQL statement, usually through the Beeline client or a JDBC connection.
- Hive plans the work and runs it as a distributed job across the cluster.
- The result comes back to your client, or is written into another table.
Where the metastore fits
The Hive Metastore holds the catalogue: which tables exist, what columns they contain and where the files live. It became widely used because it solved a shared problem, so other query engines can connect to the same metastore and see the same tables. That is why the Hive Metastore is mentioned in documentation for tools that are otherwise unrelated to Hive.
Who tends to use Hive
Hive suits teams that already run a Hadoop cluster or hold large volumes of files in distributed storage and want SQL access to them. It fits analysts and data engineers doing warehouse style work such as scans, joins and aggregations over large tables.
What Hive is not suitable for
The project is clear that Hive is not built for online transaction processing. Queries are planned and distributed, which suits large batch work rather than the fast single record lookups an application would need.
Hive also depends on how the data is laid out. A table with no partitioning forces a query to read every file, so partitioning and file format choices matter more here than in a conventional database.
Getting started with Hive
The official site publishes an installation guide, a getting started page and the HiveQL language manual. A first table can be created with a create table statement that points at a folder in storage, after which a plain select will run against it.
Most installations connect through HiveServer2 and the Beeline command line client, both covered in the documentation.
Key features of Apache Hive
Capabilities described in the official documentation.
SQL over files in storage
Hive Query Language lets you select, insert and join data that is stored as files, using syntax close to standard SQL.
The Hive Metastore
Table names, column types and file locations are kept in a central metadata store that other engines can also read.
Several storage formats
Tables can be defined over plain text, ORC, Parquet and Avro files, so existing data does not have to be converted first.
Partitioning and bucketing
Data can be split into partitions and buckets so a query reads only the files it needs rather than the whole table.
Advantages of Apache Hive
Factual advantages that follow from the features above.
SQL skills carry over
Analysts who already write SQL can query data in distributed storage without learning a distributed programming model.
One definition, many engines
Because the metastore is shared, a table defined once in Hive can be read by other engines that support it.
Built for very large tables
The project describes Hive as a system for analytics at massive scale, covering data sets that reach into petabytes.
Existing files become queryable in place
A table definition points at files already in storage, so data does not have to be loaded before it can be queried.
Common use cases for Apache Hive
Situations the official documentation describes this tool as being used for.
Reporting over a Hadoop cluster
Scheduled SQL queries summarise raw files held in HDFS or object storage into smaller reporting tables.
Sharing table definitions
The Hive Metastore holds the schema, and separate query engines point at the same definition rather than each keeping its own.
Batch transformation jobs
Long running jobs read raw partitions, apply joins and aggregations and write the result back as new partitions.
Making a landing area queryable
Tables are defined over the folders where raw files already arrive so they can be read with SQL.
Official website
Everything on this page is based on the official documentation for Apache Hive. You can read the source here.
Apache Hive official documentation