Cloudera

Why distributions exist

The Hadoop ecosystem is not one product. It is a collection of separate open source projects: distributed storage, processing frameworks, a SQL layer, a coordination service, an access control component, a metadata service, a scheduler.

Each is developed independently, on its own release schedule, by its own community.

Assembling them means answering a question that has to be answered again with every upgrade: which versions of these projects work together.

That question is more difficult than it sounds. Projects depend on each other, sometimes on specific versions, and a combination that works is discovered by testing rather than by reading documentation.

A distribution answers it once. Components are supplied together in versions tested against each other, with tooling to install and operate them and support covering the result.

For organisations that wanted the capability without a team dedicated to integration, that was a compelling proposition, and it is why distributions became the normal way large Hadoop platforms were run.

What the platform adds

Three things sit on top of the open source components.

Management. Installing, configuring and monitoring a cluster of many components across many machines is substantial work. Tooling covering all of them from one place replaces administering each project separately.

Security integration. Each component has its own notion of permissions. Integrating access control across them, so a policy applies consistently rather than being configured six times, is work the platform performs.

Support. When something fails in a system built from a dozen interacting projects, the failure is frequently in the interaction rather than in any one component. Support covering the combination is different from support for individual projects, and it is a large part of what organisations were paying for.

Where these platforms stand now

Large Hadoop platforms were built extensively during a period when they were the practical way to process datasets beyond what a single system could handle.

Cloud analytics services have since changed that calculation for many workloads. Object storage, managed query engines and processing services provide comparable capability without a cluster to operate.

This does not make existing platforms worthless, and it does explain why Cloudera appears so often in modernisation discussions rather than in new platform decisions.

Cloudera offers deployment in cloud environments as well as on premises, which the official documentation covers, and organisations should read that material directly rather than relying on descriptions of what the platform was several years ago.

What makes moving difficult

Organisations planning to move away from an established Hadoop platform consistently underestimate the same thing.

The data can be moved. Copying files from distributed storage to object storage is a tractable engineering task.

What is difficult is the processing. Years of jobs, transformations and scheduled work accumulate. They encode business logic that exists nowhere else, written by people who have left, running successfully enough that nobody has examined them.

Establishing which of those jobs still matter, what they actually do, and how to express them elsewhere is where these programmes spend their effort.

The consistent lesson is that assessment deserves more time than it usually gets. Knowing which processing is genuinely used, as opposed to merely still running, changes the size of the task substantially and is worth establishing before committing to a plan.

Who uses it

Cloudera is used by data platform teams, data engineers and operations teams in large organisations, concentrated in telecommunications, banking, insurance and government, where substantial platforms were built and continue to run.

Points to consider

Cloudera is a commercial platform and the official documentation is the reference for current offerings, components and deployment options, all of which have changed considerably over the product's history.

Operating a platform of this kind requires specialist skills. That is a genuine planning consideration for organisations depending on one over several years.

Where an existing platform serves its purpose reliably, replacing it should follow from a requirement rather than from the general direction of the industry. Migration is expensive and carries risk, and the benefit has to be specific.

For new analytics platforms, cloud services address many workloads with less operational burden, and the comparison should be made against actual requirements rather than against general positioning.

Getting started

The official documentation covers the platform's components, installation, management and security. For anyone working with an existing installation, the management and administration material is the practical starting point, since operating the cluster is where most of the work is.