Apache Kafka and event streaming

In short: Kafka is a distributed, durably stored event log. Applications append events at the end; others read them at their own pace, even hours or days later.

The difference to a classic queue

A queue usually deletes a message once it has been picked up. Kafka keeps it for a configurable time (or forever). That lets several independent readers read the same log and start from the beginning when needed.

Building blocks

• Topic: a named stream, for example orders.
• Partition: a topic is split into parts (sharding and partitioning). Within a partition the order is fixed, across partitions it is not. A message’s key (say the customer ID) decides the partition so related events stay in order.
• Offset: the running number of a message in the partition. Each reader remembers where it is.
• Consumer group: several instances of an application share the partitions. Within a group each partition is read by only one instance.
• Replication: each partition lives on several servers (replication: leader and follower); if one fails, another takes over.

Typical uses

Event streams between Microservices, collecting logs and measurements, forwarding database changes, real-time analytics. The log is also the basis for event sourcing and CQRS.

Guarantees and limits

The default is “at least once”: after a crash a message can arrive again, so processing should be idempotent. Exactly-once processing is possible with transactions but costs effort. For a small site Kafka is usually too big; a database table or a simple broker is enough.