Amazon Kinesis

What Amazon Kinesis Data Streams does

Amazon Kinesis Data Streams is a managed service for handling data that arrives continuously. Instead of collecting records into files and processing them later, applications write each record into a stream as it happens and other applications read from that stream.

AWS runs the infrastructure. There is no cluster to install, no storage to size and no replication to configure by hand.

The parts of a stream

A stream has a name and is divided into shards. Each shard is an ordered sequence of records, and the number of shards determines how much a stream can take.

Every record you write includes a partition key. Kinesis uses the key to choose a shard. That has a direct consequence worth understanding: order is preserved within a shard, not across the whole stream. If the order of a customer's events matters, the customer identifier is the natural partition key, because it sends all of that customer's records to the same shard.

Records also receive a sequence number, which is how a consumer keeps its place.

Writing and reading

Producers are anything that puts records into the stream. AWS documents several ways to do this, including the SDKs, the Kinesis Producer Library and the Kinesis Agent for files on a server.

Consumers are the applications that read records out. More than one consumer can read the same stream at the same time, each keeping its own position, so a stream used for alerting can also feed an application that archives the data.

The documentation covers consumer options including the Kinesis Client Library, which handles tracking position and sharing shards across instances, and enhanced fan out, which gives a consumer its own read throughput.

How capacity is chosen

There are two modes. On demand adjusts capacity as traffic changes and needs no shard planning. Provisioned mode has you set the number of shards, which you then increase or decrease yourself.

The choice depends on how predictable your traffic is and how much you want to manage. Both are described in the developer guide.

Retention and reprocessing

A record does not disappear once it is read. It stays for the retention period set on the stream, which starts when the record is added. During that window a consumer can go back and read earlier records again.

That is useful when a consumer application had a bug, or when a new application needs to catch up on recent history.

Who this service suits

Kinesis Data Streams suits teams already building on AWS who need to move records continuously rather than in scheduled batches. It is aimed at developers and data engineers, since using it means writing producer and consumer applications or connecting it to other AWS services.

Limitations to keep in mind

Retention is finite. Kinesis is not a permanent store, so anything that needs keeping should be written somewhere durable before the retention window ends.

Ordering guarantees apply within a shard only, so partition key choice matters. An uneven key distribution can also send far more traffic to one shard than to the others.

Getting started

The developer guide includes a tutorial that creates a stream, writes records to it and reads them back with a simple consumer. The guide also covers monitoring, capacity modes and each of the producer and consumer libraries in turn.