Event streaming engineering for Apache Kafka®

Room to grow,
in order where needed

We plan, look after and upgrade Apache Kafka® clusters, with topics and keys chosen from how each event is read, consumers watched as they keep up, and every move rehearsed on a copy first, so each app reads what it needs, at its own pace.

Thanks, we’ll be in touch soon.

{{ heroStatus }}

You’re in good company

A few of the companies we’ve worked with.

When the stream outgrew its first plan

Apache Kafka often arrives with one project. Apps write events, records that something happened, to a topic, which keeps them for a set time whether or not they’ve been read.

Each topic is split into partitions, spread across the cluster’s servers, its brokers. The apps that write are producers, and the apps that read are consumers, each keeping its own place.

Years on, topics and consumers have been added one project at a time. Nobody is sure which key each topic uses or why it has the partitions it has, some consumers fall behind at busy times, and the cluster still keeps its metadata in Apache ZooKeeper™, so every upgrade waits on a bigger move.

None of this means the setup was wrong when it was made. It means the plan stopped at the first project.

So the work starts with a look at each topic and the apps on either side of it, then changes made one at a time.

What we build

Topics planned for how they’re read, producers and consumers that keep pace, connectors watched, and clusters moved and upgraded a step at a time.

  • Topics planned for how they’re read

    Each topic’s partitions, key and retention chosen from how its events are written and read, so the events that belong together stay together.

  • Each write confirmed

    Writes confirmed by the copies they need, and retries that don’t write an event twice or change its order, as the producer settings allow.

  • Consumers that keep up

    Consumer groups, the apps that share a topic’s partitions between them, sized to those partitions, with their lag, how far each is behind the newest event, watched and the slowest step found.

  • Connectors, watched

    Data moved between the cluster and your databases and systems, each connector watched, so one that stops is noticed and started again once its cause is fixed.

  • Processing as events arrive

    Counts, joins and checks run on events as they arrive, in stream processing apps of their own.

  • The metadata move

    Clusters that keep their metadata in Apache ZooKeeper moved to KRaft, the way Kafka keeps its own metadata on its own controllers, rehearsed on a copy first.

  • Upgrades a step at a time

    Brokers, controllers and clients moved to a supported release in the order the upgrade notes give, each step checked before the upgrade is finalized.

  • Access for each app

    Encrypted connections, each app signing in as itself, and rights to read or write only the topics it uses.

  • Alerts that say where to look

    Brokers, controllers, each group’s lag and each connector watched, with alerts that say what to check first.

The key decides what stays in order

Apache Kafka keeps order within each partition: an app reading a partition reads its events in exactly the order they were written. The promise is for each partition on its own.

An event’s key, such as an order or a customer, decides its partition, so events with the same key are read in the order they were written. Events with no key are spread across the partitions.

So a key is chosen from how the events are read, never from what’s easiest to write. The narrower it is, the more evenly the events spread. The wider it is, the more stays in order together.

Partitions can be added later, but not taken away, and adding them can send a key’s new events to a different partition from its earlier ones. So each topic’s count is chosen with room to grow.

Choose what each event is keyed by, and see where it lands.

Events at Ollerby Goods

An example
Key each event by

Your order The same customer’s other order Other customers

The topic’s partitions, oldest event first

  1. First partition

    • The same customer’s other order: Placed
    • The same customer’s other order: Paid
    • Another customer: Placed

  2. Second partition

    • Your order: Placed
    • Your order: Paid
    • Your order: Shipped

  3. Third partition

    • Another customer: Placed
    • Another customer: Paid

Order status

Your order, as read

  1. Placed
  2. Paid
  3. Shipped

The topic’s partitions, oldest event first

  1. First partition

    • Another customer: Placed

  2. Second partition

    • Your order: Placed
    • Your order: Paid
    • The same customer’s other order: Placed
    • Your order: Shipped
    • The same customer’s other order: Paid

  3. Third partition

    • Another customer: Placed
    • Another customer: Paid

Order status

Your order, as read

  1. Placed
  2. Paid
  3. Shipped

The topic’s partitions, oldest event first

  1. First partition

    • Your order: Placed
    • Another customer: Placed
    • Your order: Paid

  2. Second partition

    • The same customer’s other order: Placed
    • Another customer: Paid
    • Your order: Shipped

  3. Third partition

    • The same customer’s other order: Paid
    • Another customer: Placed

Order status

Your order, as read

  1. Placed
  2. Shipped
  3. Paid

Read in a different order than written

All three in one partition, read in the order they were written.

In order for each customer. A busy customer’s events all land in one partition.

Spread across partitions, so they can be read in a different order than they were written.

Each topic’s key is chosen from how its events are read, and written down beside it.

{{ orSaid }}

A supported release, reached one step at a time

Apache Kafka aims for a new release every few months, and its bugfix releases are made only for the releases it lists as supported.

Since 4.0, Kafka keeps its metadata only in KRaft. A cluster that still keeps it in Apache ZooKeeper moves it first, on the bridge release the project names for the move: broker by broker, while the cluster keeps serving, and the move can be turned back until it’s finalized. Then the cluster upgrades.

Brokers, controllers and clients move in the order the upgrade notes give, the brokers one at a time, and the cluster is checked before each upgrade is finalized, since a finalized upgrade can’t always be taken back.

A cluster on a release that’s no longer supported gets a next step, the path to one that is, written down and tried on a copy first.

So we keep a list of each cluster’s release and how it keeps its metadata, and plan its next move while its release is still supported.

Start with one topic, and the apps on either side of it

The review covers the cluster’s release and how it keeps its metadata; each topic’s partitions, key, copies and retention; the consumer groups and their lag; the connectors and their state; who can read and write what; and what’s watched today.

The first changes come in the order we’d make them, with which move comes first, the metadata or the release, and each is tried on a copy before anything changes.

What you receive is a written review and a proposal for that first step.

Questions worth asking

  1. Can we upgrade a cluster that still keeps its metadata the older way?

    Yes, in two moves. The metadata goes to KRaft first, on the bridge release the project names for it, broker by broker while the cluster keeps serving. Then the cluster upgrades to a supported release. We rehearse each move on a copy, and check the cluster before each one is finalized, the step that can’t always be taken back.

  2. Can events stay in order?

    Yes, within a partition. Events with the same key go to the same partition and are read in the order they were written, so the key is chosen for the order the readers need.

  3. Can an event arrive twice?

    By default, a consumer can see an event again after a failure, so the apps that read are built to handle a repeat. Producer settings keep retries from writing an event twice, and transactions let a read, process and write step count once inside the cluster. Anything outside it, such as a database or an email, still needs its own check.

  4. How many partitions should a topic have?

    Enough for the consumers it needs at its busiest, with room to grow, since partitions can’t be taken away and adding them later can send a key’s new events to a different partition. We choose from how the topic is read, never from a rule of thumb.

  5. Should we run it ourselves or use a managed service?

    It comes down to what each leaves to your team: the upgrades, the watching, and who’s called when a broker stops. We’ll tell you which suits you, from what your team would rather run itself.

  6. What does it cost to work with you?

    It depends on the clusters you run and the first step you choose. Before any work starts, you get a written proposal with the scope, the price or rate, and how you pay.

    Pricing

  7. Can we hire developers for our own team?

    Yes. Tell us what you need, and within 48 hours we’ll introduce developers for Apache Kafka matched to your work, working in your time zone, and you choose who joins.

    Developers for your team

Apache Kafka, Kafka, Apache ZooKeeper, ZooKeeper, Apache and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Webair is independent of The Apache Software Foundation, and isn’t sponsored by, endorsed by or affiliated with it.

Let’s build it better

Tell us which release your cluster runs, how it keeps its metadata and where its readers fall behind, and we’ll suggest where to begin, in a short call.

Thanks, we’ll be in touch soon.

{{ ctaStatus }}