Apache Pulsar

Apache Pulsar in plain terms

Apache Pulsar is a platform for moving messages between systems. One application publishes a message to a named topic and any application that has subscribed to that topic receives it. The documentation describes Pulsar as a messaging and streaming platform, and the word streaming matters because messages are stored rather than dropped once delivered.

The design decision that shapes everything else

Most messaging systems keep serving and storage in the same process. Pulsar separates them.

Brokers accept connections, route messages and track subscriptions. They hold no message data themselves. Storage is handled by Apache BookKeeper, which runs as its own set of nodes.

The practical result is that the two concerns scale independently. Needing more capacity to serve busy clients does not force you to add storage, and keeping a longer history does not force you to add brokers.

Subscriptions decide the behaviour

A Pulsar topic can act like a queue or like a stream. Which one it is depends on how consumers subscribe:

• Exclusive. One consumer only. Anything else attempting to subscribe is refused.
• Failover. Several consumers connect, one is active and the others wait to take over.
• Shared. Messages are spread across all connected consumers, which suits work that can be processed in any order.
• Key shared. Messages are spread across consumers, but every message with a given key goes to the same consumer, so related messages stay in order.

This is why Pulsar is often described as covering both queuing and streaming. The topic does not change, only the subscription type does.

Tenants, namespaces and topics

A Pulsar cluster is organised in three levels. Tenants sit at the top, usually representing a team or a product. Each tenant holds namespaces, and each namespace holds topics.

Policies such as retention, message expiry and geo replication are applied at the namespace level. That is what makes it practical for separate groups to share one cluster without interfering with each other.

Extra pieces the platform includes

Pulsar Functions run small pieces of logic against messages on a topic and can publish the result elsewhere, without a separate processing framework.

Pulsar IO provides connectors for moving data between Pulsar and external systems.

Tiered storage moves older segments of a topic into object storage. The data remains readable through the same client interface, so consumers do not need to know where it now lives.

Geo replication copies messages between clusters in different regions, configured on the namespace.

Who Pulsar suits

Pulsar fits teams building event driven systems, and platform teams that want one messaging cluster shared by several groups. Client libraries are available for Java, Go, Python, C++, Node.js and C sharp, so applications in different languages can use the same topics.

What to consider before adopting it

Pulsar has more components to operate than a single process broker. A deployment involves brokers, BookKeeper nodes and ZooKeeper or an alternative metadata store, and all of them need monitoring.

The subscription model is also something to learn properly, because the same topic behaves quite differently depending on how it is consumed.

Trying it out

The documentation includes a standalone quickstart that runs Pulsar as a single process on one machine, which is enough to publish and consume messages from the command line. Separate guides cover Docker, Kubernetes and full cluster deployment.