Tag: Distributed Systems

Use each where it fits — utilize the features, don’t crown a winner

In 2016, we redesigned our game engines.

We were moving from a classic monolith to a modular monolith — lobby, game, transactions, and related concerns split by responsibility so each piece could scale and fail on its own. Once those modules stopped living in one process, they needed a way to talk without blocking each other on synchronous calls.

That seam was a message broker. We chose RabbitMQ.

This article is not RabbitMQ vs Kafka. It is RabbitMQ and Kafka — how we used each where it fit, utilized the features each one is actually good at, and stopped treating messaging as a single-tool religion.

The problem the redesign created

In the monolith, “send this to transactions” was often just a function call in the same process. After the split, a buy-in or a payout could not sit on the critical path of live gameplay. The game server had to publish and keep moving. The transaction server had to consume and act when it was ready.

We needed:

1. Point-to-point communication — some messages had a clear owner. One producer, one consumer path. Work had to land with the right module, not spray across the system by accident.

2. Broadcast when every consumer must act — some messages were not “pick one worker.” Every interested consumer had to see the message and do its own work.

3. Work distribution under load — for other flows, we needed many consumers sharing the load in round-robin style so no single worker became the bottleneck.

4. Survival under failure — if a consumer crashed mid-flight, the message could not vanish. It had to stay at the broker until something healthy consumed it.

5. High availability at the broker layer — if one broker node went down, the platform still had to serve. Gaming traffic does not wait for a maintenance window.

6. Serious throughput — we were designing for the order of 1 lakh (100,000) concurrent messages being consumed. This was not a toy queue for nightly jobs.

RabbitMQ mapped cleanly onto all of that.

Why RabbitMQ fit that architecture

Point-to-point where ownership was clear

For flows like game server → transaction server, we needed point-to-point messaging. Publish a command or event meant for one consumer path. Decouple the producer from the consumer’s speed. Keep gameplay off the money path.

RabbitMQ made that natural: publish to an exchange, bind a queue to the module that owns the work, consume with acknowledgements. The producer does not care whether the consumer is momentarily slow. The broker buffers.

Topic when every consumer must act

Not every message was “give this job to one place.”

Some messages needed every interested consumer to consume and act. For those, we used topic exchanges. Each consumer (or each consumer type) bound its own queue with the routing pattern it cared about. One publish, many independent reactions.

That gave us fan-out of intent without hard-coding a list of callers inside the producer. New consumers could subscribe by binding — the publisher stayed dumb about who was listening.

Fanout for queue round-robin under load

Where the job was pure work distribution — many workers, same kind of work, share the load — we used fanout into queues that consumers competed on in round-robin fashion.

One message → one worker. Add more consumers, get more parallelism. That pattern is RabbitMQ at its most boring and most useful: a durable work queue in front of a pool of processors.

Between topic (everyone who cares acts) and fanout / competing consumers (one worker acts), we covered both “notify the system” and “do the job once.”

Durable queues: messages stay until consumed

This was non-negotiable in a gaming stack that touched money and live state.

We used durable queues so that:

Queues survived broker restarts.

Messages were not treated as ephemeral fire-and-forget.

If a consumer broke, the message stayed at the broker until a healthy consumer took it and acknowledged it.

That durability model matched how we thought about reliability: the broker is the safety net between modules. A crash in one modular service must not erase in-flight work for another.

In practice that meant designing for at-least-once delivery. Consumers had to be idempotent. Duplicates can happen when a consumer dies after doing the work but before the ack. Durable queues protect you from loss. They do not invent exactly-once magic. We accepted that trade-off consciously.

Dead letter queues for transactional outages

Durable queues keep messages alive. Dead letter queues (DLQs) gave us a place to put work that could not complete cleanly.

When a transaction path hit an outage — consumer errors, rejected messages, retries exhausted, or a downstream money path temporarily unhealthy — those messages did not disappear into a black hole. They landed on a dead letter queue, where we could inspect them, retry them, or run a controlled recovery once the outage cleared.

For gaming, that mattered as much as the happy path. Buy-ins and payouts cannot be “lost because the consumer threw.” DLQs turned transactional failure into something operable: quarantine, diagnose, execute again when the system was ready.

So the reliability story had two layers:

Durable queues — message stays until a healthy consumer acknowledges it.

Dead letter queues — failed or unprocessable transactional work stays available for recovery after outages.

Absolute match: act in milliseconds, not replay history

For our game engines, RabbitMQ was not “good enough.” It was an absolute match for what a broker had to do.

Gameplay and money-adjacent flows had to act within milliseconds. The job of the broker was to get the right message to the right consumer fast — point-to-point, topic, or round-robin — and keep that path reliable under failure. We did not need to save a long history of every message so someone could replay last week. Once the work was done (or parked on a DLQ for recovery), the broker had finished its job.

That distinction is easy to miss when you evaluate messaging tools in the abstract. Game engines live in the present tense. Publish, route, consume, act. RabbitMQ is built around that shape of work, and in production it worked fine for us at the load and availability bar we needed.

High availability: keep serving when a node dies

Durability at the queue level was only half the story. The other half was broker high availability.

We ran RabbitMQ as a distributed cluster. The requirement was simple and absolute: if one node went down, the system should keep serving. Producers and consumers had to continue — buy-ins, payouts, and cross-module work could not freeze because a single broker machine failed.

That is where RabbitMQ’s distributed mechanisms mattered for us:

Queues and messages were set up so work survived node loss, not only process crashes on the consumer side.

Clients could reconnect and resume against healthy nodes.

The cluster absorbed a node failure without turning the modular game engines back into a tightly coupled outage.

Distributed brokers also force you to think about network partitions and split-brain — the ugly cases where nodes cannot agree on who is in charge. For our gaming requirements, we operated the cluster so those distributed behaviors stayed within what the platform could tolerate: failover worked, serving continued, and we did not see the broker layer become the weak link when a node disappeared.

In a live gaming environment, “high availability” is not a slide. It is whether tables keep moving and money paths keep accepting work at peak. RabbitMQ’s HA model met that bar for us.

Scale: 1 lakh concurrent messages consumed

People sometimes dismiss RabbitMQ as “fine for small systems.” That was not our experience.

Configured and operated carefully — durable queues, dead letter queues, HA across nodes, sensible prefetch, enough consumers, monitoring on queue depth and consumer lag — RabbitMQ handled the load we needed: on the order of 100,000 concurrent messages being consumed across the paths that mattered for the modular game engines.

We did not pick it because a blog said it was trendy. We picked it because our patterns were route and act in milliseconds, with availability under node failure and **DLQs for transactional outages — and the broker could take the concurrency we were aiming at.

Then the business grew: analytics and BI

As the business grew, analytics and BI arrived with a new class of requirements.

They needed game events and transactional events stored in the BI system — routinely — so the business could analyze what players did and how money moved. Data lakes and warehouses entered the conversation. Snowflake and similar systems became the place where transformed events were supposed to live for reporting and analysis.

Our first instinct was natural, and a little dangerous: we already had RabbitMQ in the middle of the game engines. So we suggested RabbitMQ again.

The plan was simple on paper:

1. BI / analytics consumers bind and consume from the relevant exchanges.

2. They transform the messages.

3. They store the results into the BI system / data lake (for example Snowflake).

For some days, that also worked fine. Events flowed. Transforms ran. Rows landed in the warehouse. It looked like we had extended the same broker to a new audience without inventing new infrastructure.

That comfort did not last.

The day risk analysis changed the question

One fine day, the risk analysis team came with a harder ask.

They did not only want a live feed into BI. They wanted to transform events in ways the BI team could modify — evolving logic, re-shaping pipelines, changing how raw game and transactional events became analytical facts. And they needed the underlying messages available for reconciliation the next week.

That sentence is where the architecture cracked.

RabbitMQ had been perfect when the job was: deliver this work now, act in milliseconds, then you are done.It is a weak fit when the job becomes: keep the events around, let multiple teams re-transform them, and let risk come back next week to reconcile against what actually happened.

We had stretched a work broker into an event history problem.

Once a message is consumed from a classic queue path, it is not sitting there as a week-long source of truth for reconciliation. DLQs help with failed processing. They are not a BI-grade archive of every game and transactional event for next week’s risk checks. Asking every new team to “just consume from the exchange” also couples analytical evolution to the same live routing fabric the game engines depend on — and it does not give risk a clean way to re-read last week’s facts after BI changes a transform.

That is when it became clear: for this case, we had chosen the wrong broker.

Not wrong for game engines. Wrong for analytics + BI + risk reconciliation over time.

The requirements had shifted from act now to retain, re-transform, and reconcile later. Same events. Different job. Different tool.

Introducing Kafka for BI and analytics

So we introduced Kafka for the BI / analytics path.

Not as a replacement for RabbitMQ on the game engines. As the right backbone for a different job: game events and transactional events that must be stored, analyzed as routine, re-transformed when BI logic changes, and still available when risk comes back next week for reconciliation.

Kafka for BI and analytics: game and transactional events plus Cassandra CDC feed a commit-log cluster; transform consumers reshape events in transit; BI, risk reconciliation, and other analytics read independently into the data lake.

What Kafka gave us that RabbitMQ was not designed to be for this case:

A commit log, not a disposable queue

Kafka is built as a distributed commit log. Events are appended to topics and retained by policy. Consumers track an offset — a position in history — instead of “take this message and it is gone from the shared truth.”

That is the mental model analytics needed. The stream is a durable record of what happened. Risk can return to last week’s events. BI can change a transform and re-read. The log stays.

History you can replay

For reconciliation and evolving analytics, history is the feature.

With Kafka, game and transactional events remain available for the retention window we set. A consumer that was wrong on Monday can be fixed on Thursday and replay. A risk job that needs last week’s facts does not depend on someone having saved a side copy “just in case.” The platform keeps the events.

That is exactly what broke when we tried to stretch RabbitMQ: once the live consume path had moved on, next week’s reconciliation had no first-class history to stand on.

Transformation in transit

BI and risk did not only need raw events parked forever. They needed message transformation in transit — shape the event on the way into lakes and warehouses (Snowflake and similar), and keep the ability to change that shaping as analytical definitions evolved.

Kafka fits that pipeline shape:

1. Producers (or bridges from the game / transaction world) publish canonical events to topics.

2. Stream / transform consumers read the log, apply BI-owned logic, and write curated results downstream.

3. When the transform changes, teams can redeploy the transform and, where needed, replay from history instead of begging for a one-off re-extract.

4. Risk can consume the same underlying topics independently for reconciliation — without stealing messages from the BI consumer or coupling to the game-engine RabbitMQ fabric.

The important architectural point: **transformation becomes a consumer concern on top of a retained log**, not a one-shot side effect of emptying a work queue.

CDC from Cassandra into Kafka

Kafka also fit another reality of our stack: we already had important state in Cassandra.

For analytics and BI, it was not enough to stream only the messages the game engines happened to publish. We needed changes from Cassandra tables themselves — inserts and updates that represented durable game and transactional facts available as an event feed.

So we integrated Cassandra → Kafka as CDC (change data capture). Table changes were captured and published into Kafka topics. From there, the same commit-log strengths applied: retain history, transform in transit, land curated data in the BI lake, and let risk reconcile later by re-reading the stream.

That pattern worked well. CDC turns the database into a producer of facts without forcing every service to remember to emit a perfect analytics event on every write. Kafka is a natural sink for that feed because it is built to hold an ordered, replayable log of changes — not to empty a work queue and forget.

Use each where it fits

After Kafka landed for BI and analytics, we were not declaring a winner. We were **using each technology where it fit** and utilizing the features that matched the job:

We did not “upgrade from RabbitMQ to Kafka.” We utilized RabbitMQ for work routing and Kafka for the commit-log / history path. Both stayed. Both earned their place.

Where it fit	What we utilized
Game engines: route work, act in milliseconds	RabbitMQ — exchanges, durable queues, DLQs, HA
BI / analytics / risk: history, transform, reconcile later	Kafka — commit log, replay, transform in transit
Cassandra table changes into the analytical stream	Kafka CDC

What I’d tell a team building the same platform

Do not start with “which technology is better.” Start with the job, then utilize the features that fit:

1. Point-to-point? You need clear ownership of work — RabbitMQ exchanges and queues fit well.

2. Every consumer must act? Topic-style fan-out to many queues.

3. Round-robin workers? Competing consumers on shared work queues.

4. Consumer can die? Durable queues and ack-based consumption so messages stay until processed.

5. Transaction path can fail?Dead letter queues so outages become recoverable work, not silent loss.

6. Broker node can die? A real HA cluster — including partition / split-brain behavior, not only the happy path.

7. Act in milliseconds, no long history? A classic broker like RabbitMQ is often the cleaner fit — that was our game-engine case.

8. Analytics, BI, or risk need events next week — to re-transform or reconcile? Utilize a commit log with history — for us, that was Kafka, including CDC from Cassandra.

We asked the first set of questions in 2016. RabbitMQ fit. We utilized it.

We asked the later questions when the business grew. Kafka fit. We utilized it.

That is the whole lesson.

Wrapping up

Do not compare technologies to pick a champion. Use them wherever they fit. Utilize their features.

RabbitMQ fit our modular game engines: point-to-point, topic broadcast, fanout with round-robin, durable queues, dead letter queues, HA, and roughly 1 lakh concurrent messages consumed— act within milliseconds.

Kafka fit BI, analytics, and risk: a commit log, history for next-week reconciliation, transformation in transit into lakes like Snowflake, and CDC from Cassandra tables.

That is why this article is named RabbitMQ and Kafka — not RabbitMQ vs Kafka.

Same company. Same events. Different jobs. Both tools, used on purpose.