Tag: Software Architecture

Every Architecture Decision You Skip Recording, You Pay For Twice

What missing ADRs cost when you build a Dynamic Real-Time Query Engine — not Slack archaeology

When a new requirement lands, we open the Architecture Review Board (ARB).

Someone says “buy another CRM tool.” Someone says batch is fine. Someone says “we already decided this.” Nobody can find where.

That is not a tooling problem. It is a memory problem — and on a live gaming platform, forgotten decisions become production incidents, re-opened fights, and sprints that undo last year’s work.

Architecture Decision Records (ADRs) are how you stop paying twice. Not bureaucracy. Insurance.

This article is not “what is an ADR” from a textbook. It is a consequence map: decisions we made on a real gaming platform — Dynamic Real-Time Query Engine for CRM and ML, lobby and live tables, wallets, caches, fraud on the hot path — and what broke when we did not record them (or recorded them in Slack and lost them).

ADR in one sentence (before the horror stories)

An ADR is a short, durable note: what we decided, why, what we rejected, and what we will revisit. It lives in the repo or architecture wiki — not in a thread that scrolls away.

You do not need one ADR per line of code. You need one per decision that will be fought again if nobody wrote it down.

The decisions that deserved an ADR (gaming platform)

On our platform, these were not “nice to have.” They were forks in the road:

Decision	Why it mattered
RTQE as platform, not CRM patch (Cassandra + JSON DSL + shared API)	Millisecond behavioral queries for CRM and ML — one live truth
JSON DSL vs open SQL on RTQE	CRM self-serve segments without unbounded queries
REST for lobby, WebSocket for live play	Door vs room — different conversation shapes
Hazelcast for table state, Redis for leaderboards	CAP — coordination vs eventually-fresh reads
One writer per ledger	No duplicate wallet paths during strangler migration
Fail-open vs fail-closed on fraud	Block play vs allow play when the engine is down
Reuse Spark / buy EMR vs build segmentation	Cost and ownership on offline compute
Strangler order: transactions before lobby split	Plateau sequence — not big-bang weekend

Each of these showed up in ARB, stand-up, or a new squad’s first design doc. When we missed recording one, we paid a predictable price.

If you miss the ADR — consequences (with examples)

1. Dynamic Real-Time Query Engine — “let’s bolt another tool onto CRM”

Decision: Build the Dynamic Real-Time Query Engine (RTQE) as shared platform infrastructure — not a one-team patch. Cassandra time-series (one table per event type), JSON DSL for behavioral segments, REST APIs for CRM, targeting, and the ML Service — millisecond queries while the player is still on screen.

If you skip the ADR:

CRM buys or builds a narrow segmentation tool; ML spins up a second live-events pipeline.

Consequence: duplicate stores, segments that lag live play, engineering tickets for every new behavioral question — “real-time” becomes tomorrow’s batch.

Symptom in prod: offer fires after the player left the table; CRM and ML disagree on the same player’s last hour.

What the ADR should have said: Platform not patch. One engine, many consumers. Bounded JSON DSL — not open SQL. Cassandra model: player_id partition, time-series clustering, one table per event type.

2. RTQE query model — “just expose SQL to CRM”

Decision: JSON DSL with bounded query models (aggregates, time windows, expressions) — every query maps to a single-partition, time-bounded Cassandra read. Not open SQL to marketing tools.

If you skip the ADR:

Someone exposes read-only SQL “for flexibility.”

Consequence: unbounded scans, hot partitions, on-call pages during a campaign launch — CRM “self-serve” becomes production roulette.

Symptom in prod: segment job takes 30 seconds; player already churned; CRM blames “the real-time engine.”

What the ADR should have said: DSL not SQL. Query bounds by construction. `searchTime` aligns with time-series clustering key.

# ADR-001: Dynamic Real-Time Query Engine as shared platform (not CRM patch)

Status: Accepted  
Date: 02/15/2025
Deciders: Architecture Board, CRM, Data Platform, ML, VP Engineering  

Context

CRM and marketing needed to understand live player behavior and act while the player is still on screen  not via nightly batch jobs. Existing CRM tools could not answer behavioral questions in milliseconds. The easy path was another point solution on the CRM stack; the harder path was shared platform infrastructure.

 Decision

Build the Dynamic Real-Time Query Engine (RTQE) as a foundational building block:

 Apache Cassandra — one table per event type, time-series keyed by `player_id` + event timestamp
 JSON DSL — bounded behavioral queries (aggregates, time windows, expressions) — not open SQL
Event ingest pipeline — decoupled producers into the engine (point-to-point queues)
 Java 21 + Spring Boot — REST APIs for CRM, targeting, and ML Service consumers
Platform scope — same engine serves multiple consumers — not a single-team patch

Consequences

Positive: Millisecond segments on live play; CRM self-serve in JSON; ML gets real-time features from one pipeline; ~100K req/s headroom with virtual threads and horizontal scale.

Negative: Custom DSL to maintain; Cassandra modeling discipline required; platform team owns SLAs for all consumers.

Obligations: Query bounds by construction; one-table-per-event-type schema governance; document rejected “SQL over everything” and “CRM-only tool” paths.
| Alternative | Why rejected |
|-------------|--------------|
| Bolt-on CRM segmentation tool | Solves one team;  duplicate pipelines for ML; batch mindset |
| Expose raw SQL to CRM | Unbounded queries; safety and ops risk on hot path |
| Single generic events table | Partition and query unpredictability; fights Cassandra strengths |
| Nightly batch only | Misses the conversion window — player already left |

3. REST lobby + WebSocket game — “let’s poll matchmaking from the socket”

Decision: REST (or HTTP API) to Lobby — find and join a table. WebSocket to Game — live play, broadcast state.

If you skip the ADR:

Mobile team polls lobby state over the game socket every 500ms for simplicity.

Consequence: socket fan-out load explodes; matchmaking logic bleeds into game shards; you cannot scale browse traffic separately from seated players.

Symptom in prod: lobby spike during a tournament drags down live tables — the failure domain you split on purpose.

What the ADR should have said:Door = REST. Room = WebSocket. Do not merge because one client library is easier.

# ADR-002: REST for lobby (door), WebSocket for game (room)

Status: Accepted  
Date: Migration Plateau 2  
Deciders: Mobile, Lobby, Game squads, ARB  

Context

Matchmaking is request/response: find table, seat player, return connection info. Live play is a long-lived session: broadcast state, play events, timers. Mixing both on one channel couples browse traffic to seated-player scale.

## Decision

REST (HTTP API) to Lobby Service for matchmaking, presence, seat assignment.
- WebSocket to Game Service for live table play after seat confirmed.
- Session handoff: lobby returns game shard / token; client opens socket to game fleet.

## Consequences

Positive: Scale lobby on browse traffic independently; clear failure domains; matches client mental model (door vs room).

Negative: Two client integrations; handoff bugs if token/session contract drifts.

Obligations: Document handoff flow in API spec; integration tests for seat → connect path.

 Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| WebSocket for lobby polling | Fan-out and unnecessary persistent connections for short requests |
| REST for live play | Polling latency unacceptable for table state; wrong conversation shape |
| Single monolith socket | Failure domain and scale coupling we were leaving |

4. Hazelcast vs Redis — the annual CAP debate

Decision: Hazelcast (remote cluster, sync backup, CP locks) for recoverable table state when a game node dies. Redis (cluster) for leaderboards where slightly stale ranks are acceptable.

If you skip the ADR:

Every new hire proposes Redis for everything or Hazelcast for “leaderboards too.”

Consequence: quarters lost to benchmarks and slide decks; wrong tool on wrong workload; wallet or pot state in a cache that was never meant to be the ledger.

Symptom in prod: failover restores the wrong pot; leaderboard shows ranks from before a partition.

What the ADR should have said:HZ = coordination + recovery (lean C). Redis = high-read scores (lean A). Money still flows queue + DB.

# ADR-003: Hazelcast for table recovery, Redis for leaderboards

Status: Accepted  
Date: Post modularization scale-up  
Deciders: Platform ARB, Game squad  

 Context

Game nodes are pinned per table with local active state. On node death, we must recover pot, seats, and turn without split-brain. Leaderboards are high-read, tolerate slight staleness, and do not need CP semantics for every rank update.

 Decision

Remote Hazelcast cluster (six members, clients only from game/lobby services): `IMap` with sync backup for table snapshots; CP locks + fencing on failover.
Redis cluster (three nodes) for leaderboards reads/writes — eventual consistency acceptable for ranks.
Wallet ledger stays queue + database — not in either cache.

 Consequences

Positive: Correct recovery for live money tables; leaderboard scale without HZ partition pressure on every rank tick.

Negative: Two cache platforms; clear ownership required per workload.

Obligations: Failover drills; monitor backup lag; no dual writers on game state.

Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| Redis for table state | Wrong CAP fit for pot recovery without careful fencing; risk of split brain |
| Hazelcast for leaderboards | Overkill; CP cost not justified for ranks |
| Embedded HZ in game JVM | Blast radius; fleet churn reshapes grid |

5. One writer per ledger — “we’ll dual-write during migration”

Decision: During strangler migration, only Transaction Service writes the wallet ledger. Monolith wallet path must decommission, not linger “just in case.”

If you skip the ADR:

Plateau 1 ships with two writers — monolith and new service both mutate balance.

Consequence: race conditions, support tickets (“where did my chips go?”), reconciliation nightmares, audit failure.

Symptom in prod: player balance disagrees with ledger after a hand — the gap ArchiMate was supposed to close.

What the ADR should have said:One writer per ledger. Coexistence duration named. Decommission rule in the same doc as the migration ADR.

 ADR-004: One writer per wallet ledger during strangler migration

Status: Accepted  
Date: Plateau 1 (transactions strangler)  
Deciders: Transactions squad, Platform ARB, Risk  

 Context

Extracting Transaction Service while the monolith still serves lobby and game creates a coexistence window. Dual wallet paths are tempting for “safety” but produce race conditions and audit gaps.

 Decision

Transaction Service is the sole writer to the wallet ledger after cutover.
Monolith in-process wallet mutation 
decommissioned by named date — not left dormant. Coexistence: read paths may dual-serve briefly; writes do not.

 Consequences

Positive: Auditable ledger; support can trust one source of truth; strangler gap closes cleanly.

Negative: Requires feature flags and migration testing; rollback plan must be explicit.

Obligations: ADR linked in ArchiMate gap P0→P1; reconciliation job until decommission complete.

Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| Dual-write monolith + service | Race conditions; “where did my chips go?” tickets |
| Big-bang cutover weekend | Unacceptable downtime risk for live tables |
| Ledger in game cache | Violates money-off-hot-path principle |

6. Fail-open vs fail-closed — fraud on the hot path

Decision: On player-facing blocks, prefer explainable decisions and clear audit. On hot-path latency, define what happens when the fraud engine is unavailable — fail – open vs fail-closed is a business choice, not a default in the SDK.

If you skip the ADR:

On-call flips behavior under pressure without a recorded principle.

Consequence: either legitimate players frozen at peak (revenue loss) or abuse runs unchecked during an outage (loss + trust hit).

Symptom in prod: incident post-mortem says “we thought prod was fail-closed” — staging was fail-open.

What the ADR should have said:Gameplay path: [X]. Wallet path: [Y]. Owner: Risk + Platform. Review when SLA changes.

# ADR-005: Fail-open vs fail-closed on the fraud hot path

Status: Accepted  
Date: Fraud engine Phase D (Technology Architecture)  
Deciders: Risk, Payments, Platform ARB, Game squad  

 Context

The real-time fraud engine sits on game and payment hot paths. When the Decision API is slow or unavailable, someone must decide: block traffic (fail-closed) or allow traffic (fail-open) until recovery. That is not an SDK default — it is a business and compliance choice. Without a recorded policy, staging, prod, and on-call behave differently.

 Decision

Define a policy matrix by path and transaction type — documented, tested, and owned by Risk + Platform:

| Path | Default when engine unavailable | Rationale |
|------|----------------------------------|-----------|
| Gameplay / join table | Fail-open with rate limits + post-hoc review | Revenue and player experience at peak; friction on outage freezes live tables |
| Wallet / buy-in / payout | Fail-closed (or queue + async settle with hold) | Money movement requires audit; abuse during outage is unacceptable |
| High-value withdrawal | Fail-closed always | Chargeback and regulatory exposure |

Cross-cutting rules:

Explainable reason codes required for any player-facing block when engine is healthy  
Shadow mode in migration — score beside legacy; do not enforce until false-positive SLA holds  
Break-glass override with Risk RACI — not an undocumented on-call toggle  
Same matrix in staging and prod — no silent drift  

 Consequences

Positive: Incidents have a playbook; no “we thought prod was fail-closed”; ARB can govern Phase G against a written matrix.

Negative: Fail-open on gameplay accepts temporary abuse risk — must pair with velocity caps, DLQ review, and post-incident replay.

Obligations: Latency budgets measured; policy matrix in runbooks; ADR linked from Decision API config and ArchiMate technology constraints.

Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| One global fail-closed | Legitimate players frozen at peak; revenue loss during engine blip |
| One global fail-open | Wallet abuse during outage; audit and trust failure |
| On-call decides per incident | Inconsistent behavior; post-mortem blame without policy |
| Hard-code in payment service only | Game path undocumented; six squads, six truths |
| Vendor SDK default | Business policy hidden in library; not governed by ARB |

7. Build vs buy vs reuse — “let’s build our own Spark”

Decision: Reuse Apache Spark for offline segmentation; buy managed EMR where the bill fit; build only the thin control plane (scheduling, tenancy, guardrails) — not a second distributed compute engine.

If you skip the ADR:

A squad starts a 10-month “lightweight batch engine” because EMR “felt expensive” on one slide.

Consequence: duplicate platform, no hiring market, security review from scratch — savings illusion.

Symptom in prod: two batch systems, neither owned; CRM segments diverge from real-time truth.

What the ADR should have said:Reuse Spark. Buy EMR when TCO wins. Build only orchestration we cannot buy.

# ADR-006: Build vs buy vs reuse for offline segmentation (Spark)

Status: Accepted  
Date: Phase E (Opportunities & Solutions) — after RTQE platform  
Deciders: Platform ARB, Data Platform, CRM, Finance  

 Context

CRM needed offline behavioral segments that aligned with the Dynamic Real-Time Query Engine — same player truth, batch scale for campaigns and analytics. The commodity capability is distributed batch compute. The enterprise question was not “Spark or not Spark” but reuse the engine, buy managed EMR, or build a custom runtime.

 Decision

Compose — do not pick one label for everything:

| Choice | What we chose | What it means |
|--------|---------------|---------------|
| Reuse | Apache Spark | Offline segmentation engine-  commodity distributed compute; do not reinvent |
| Buy (evaluated) | Amazon EMR — not selected for this workload | EMR scales; for our utilization pattern the bill was ~10x a lean in-house runtime |
| Build | Thin control plane only | Spark master on EC2 (published IP); worker pods autoscale; custom scaler (5‑min warm-up, min/max per job); ~10 days to deliver |

Rules recorded:

Reuse what is commodity (Spark). Buy when TCO and operating model both fit. Build only the layer that turns reuse into enterprise fit — not a second batch engine.
Offline segments must trace to the same event semantics as RTQE — no shadow data estate for CRM.
Owner: Data Platform operates runtime, scaler, and upgrades.

 Consequences

Positive: ~10× lower ongoing cost vs EMR path for our pattern; delivery in ~10 days; CRM segments stay aligned with real-time truth.

Negative: We own Spark master failover, scaler logic, and security patches — not EMR’s managed boundary.

Obligations: Revisit EMR when utilization or compliance changes; document payback in ARB; no “build our own Spark” proposals without new context.

 Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| Buy EMR for this workload | Capability fit yes; cost fit no (~10× bill for our utilization) |
| Build a new batch engine from scratch | Enterprise malpractice; no hiring market; years of maintenance |
| CRM vendor segmentation module only | Duplicate pipeline; diverges from RTQE player truth |
| Reuse Spark with no control plane | Could not meet warm-up, per-job limits, tenancy we needed |
| Buy only — no reuse/build split | Treated “buy” as slogan; ignored thin-build payback |

Strangler order — “let’s split lobby first, it’s easier”

Decision: Transactions module first via broker — money off the monolith hot path. Lobby + game split in Plateau 2. Sequence documented in migration model and ADRs.

If you skip the ADR:

Product pushes lobby microservice first because the UI team is ready.

Consequence: pretty services, money still in the monolith — the riskiest coupling untouched; false sense of “we migrated.”

Symptom in prod: “we’re on microservices” but a stuck wallet call still kills live tables.

What the ADR should have said:Plateau 1 = transactions strangler. Plateau 2 = REST lobby + WebSocket game. Link to ArchiMate gap.

ADR-007: Strangler migration order — transactions before lobby split

Status: Accepted  
Date: ADM Phase F (Migration Planning)  
Deciders: Platform ARB, Product, Game, Transactions, Lobby squads  

Context

Moving from a socket monolith (lobby + game + wallet in one deployable) to modular services tempts teams to split what is easiest first — often lobby/UI — while money stays in the monolith. That creates a false "we migrated” story and leaves the highest-risk coupling (gameplay tick + wallet) untouched. Migration order is an architecture decision, not a sprint convenience.

 Decision

Strangler sequence — documented in ArchiMate plateaus and ADRs:

| Plateau | Work package | What ships | Monolith still owns |
|---------|--------------|------------|---------------------|
| Plateau 0 | — | Baseline monolith | Lobby, game, wallet (all in one) |
| Plateau 1 | WP1 — Extract transactions | Broker + Transaction Service; one ledger writer | Lobby + game |
| Plateau 2 | WP2 — Split lobby and game | REST lobby + WebSocket game handoff | — (target modular) |
| Later | WP3+ (optional) | Shard game fleet, containerize, autoscale | Per roadmap |

Rules recorded:

1. Plateau 1 = transactions strangler first — money off the gameplay hot path before cosmetic service splits.
2. Plateau 2 = REST lobby + WebSocket game — only after Plateau 1 coexistence rules and ledger ADR (ADR-004) are met.
3. No big-bang weekend — each plateau runs in production with named gaps and decommission dates.
4. Product priority does not override sequence without new ARB context (supersede this ADR).

Consequences

Positive: Riskiest coupling addressed first; live tables keep running; board can recognize plateau transitions; aligns with ADM Phase F and Agile increments.
Negative: Lobby/UI teams wait for Plateau 2; requires patience and visible migration model so squads do not fork private diagrams.

Obligations: ArchiMate work packages link to ADR-004 (one writer) and this ADR; sprint reviews ask “which plateau does this close?”

 Alternatives considered

| Alternative | Why rejected |
|-------------|--------------|
| Lobby microservice first (UI ready) | Money still in monolith; stuck wallet call still kills live tables |
| Big-bang cutover | Unacceptable downtime; no coexistence learning |
| Game split before transactions | Money path remains on hottest failure domain |
| Target-state-only migration (skip Plateau 1) | Poster architecture; production never matches model |
| Parallel all services at once | Six squads, three buses, no decommission plan |

The pattern when ADRs are missing

What you feel	What is actually happening
“We already decided this”	Decision lived in Slack / a meeting / one senior’s head
“Why did they build it that way?”	No rejected-options trail — looks like incompetence, was often a tradeoff
“New squad, same fight”	Organizational amnesia — cheaper to write 1 page than rerun ARB
“Rollback panic”	Nobody remembers coexistence rules from migration
“Architecture is slow”	You are re-deciding instead of referencing

ADRs do not slow you down. They buy back the quarters you lose re-fighting “batch CRM vs real-time platform” — or Hazelcast vs Redis.

What to put in an ADR (minimum viable)

1. Title — e.g. `ADR-001: Dynamic Real-Time Query Engine as shared platform (not CRM patch)`

2. Status — Proposed | Accepted | Superseded

3. Context — pressure, constraints, plateau you are in

4. Decision — one clear paragraph

5. Consequences — positive, negative, what we owe ops/security

6. Alternatives considered — what you said no to and why

Length: one to two screens. If it reads like a thesis, nobody will write the next one.

Where it lives: `docs/adr/` in the repo, or wiki linked from the ArchiMate work package and the Jira epic. Same decision, three doors — one truth.

ADRs × ADM × Agile × ArchiMate

Practice	Role
ADM Phase G	Governance — decisions are controlled, not oral tradition
Agile	ADR per meaningful increment — not per sprint theater
ArchiMate	Work packages link to ADR IDs — gaps close with named decisions
ARB	Accept or reject ADRs; do not re-debate without new context

Anti-patterns (ADR edition)

Slack as ADR — searchable until it is not; context dies on scroll

ADR after the outage — post-mortem theater, not architecture

ADR per README paragraph — team stops writing them

No “superseded”— two contradictory ADRs, both “accepted”

ADR with no rejected options— future you cannot defend the call

Architecture PDF instead of ADRs — pretty, disconnected from backlog

Checklist: does this decision need an ADR?

1. Will a new squad ask “why?” in six months?

2. Did we reject something credible?

3. Does it affect money, play, or compliance on the hot path?

4. Does it sequence migration (plateau, coexistence, decommission)?

5. Will ops need to know failover behavior?

If yes to two or more — write the ADR before merge, not after the incident.

Wrapping up

On a live gaming platform, every architecture decision you skip recording, you pay for twice: once when you make it, again when someone unknowingly unmakes it.

RTQE as platform. REST vs WebSocket. Hazelcast vs Redis. One writer per ledger. Fail-open vs fail-closed. Build vs buy vs reuse. Strangler order.

None of those are secret wisdom. They are decisions — and decisions without ADRs become mysteries, then incidents, then quarters in ARB.

Record the call. Name what you rejected. Link it to the plateau.

That is EA doing its job: memory over slogans.

Note on the sample ADRs in this article (ADR-001 through ADR-007 — RTQE platform, REST/WebSocket, Hazelcast/Redis, one ledger writer, fail-open/fail-closed, build vs buy vs reuse, strangler order): they are illustrative mocks — written in real ADR shape to show context, decision, rejected options, and consequences. They are not the literal records from our Architecture Review Board. In production, your ADRs live in your repo or wiki, carry your IDs, dates, deciders, and may differ in detail — use the samples as templates, not as copies of our internal governance pack.

September 13, 2026

ADM and Agile Complement Each Other

Marry the map and the sprint a live gaming platform shows how.
A lot of people confuse ADM and Agile.

Sometimes TOGAF practitioners ignore Agile as if a good architecture slide deck ships itself. Sometimes Agile warriors ignore TOGAF as if sprints alone will sort out migration, data ownership, and enterprise governance.

Both camps lose.

If you marry both you get something better standard, beautiful results not perfect, but repeatable. Architecture that names what to change and in what order. Delivery that proves each slice before the next. On a live gaming platform, that marriage is not academic. Players, money, and latency do not wait for your methodology debate.

This article is ADM and Agile not vs. How they complemented each other when we moved from a monolith toward modular game engines, matchmaking and WebSockets, brokers, real-time query, and scale using the Architecture Development Method for the spine and Agile for the heartbeat.

Two words people treat as enemies.

	ADM (TOGAF)	Agile
Answers	Why change? Capabilities? Migration? Govern?	What ships this sprint? Inspect and adapt?
Rhythm	Iterative architecture cycle (phases A–H)	Iterative delivery (backlog → sprint → retro)
Risk when alone	Beautiful target state, nothing in production	Fast features, duplicate buses, no migration map
Together	ADM sets boundaries and sequence; Agile lands increments

One line:ADM is the map and the sequence. Agile is the engine. Gaming needs both.

Why a gaming platform is a honest test.

High-scale gaming and payments punish half-methods:

Hot paths — live tables, matchmaking, wallets — cannot wait for a yearly architecture review

Many squads— lobby, game, transactions, platform — need a shared map or they invent private truths

Migration is real— monolith → modules → brokers → analytics — not a greenfield fantasy

Change never stops— new abuse patterns, new playlists, new scale events

If ADM only works on calm HR portals, it is not Enterprise Architecture. If Agile only works on a single greenfield app, it is not enterprise delivery. A live game platform is where the marriage shows its value.

ADM phase	Agile practice	Gaming example
Preliminary	Working agreements, ADRs, DoR	Money off hot path; one writer per table; door vs room
Phase A — Vision	Epic, product goal	Modular monolith — lobby, game, transactions independent
Phases B–D	Spikes, sprint zero	REST matchmaking → WebSocket play; RabbitMQ off money path
Phase E	PI planning, prioritized backlog	Kafka when analytics needs history; Spark build vs EMR buy
Phase F — Migration	Incremental release, flags	Strangler off monolith — transactions module first
Phase G — Governance	Retro, ADRs, ARB	Hazelcast + Redis CAP choice recorded once
Phase H — Change	Continuous improvement	Fraud, RTQE consumers, fleet scale — cycle repeats

Preliminary + team working agreements

ADM:principles, scope, who owns architecture decisions.

Agile: Definition of Ready, squad charters, architecture decision records in the backlog.

Gaming example: principles we actually enforced — money off the gameplay hot path; one writer per live table; matchmaking as door, WebSocket as room; reuse enterprise events before shadow copies.

Without this marriage, every squad optimizes locally. RabbitMQ in one place, Kafka in another, REST polls for the game loop and nobody owns the seam.

Phase A (Vision) + epics

ADM:business drivers, stakeholders, target state in plain language.

Agile: product goal, epic, measurable outcomes per quarter.

Gaming example:Modular monolith so lobby, game, and transactions scale and fail independently without losing live play.” Not “microservices because Netflix.”

Phases B–D (Business, IS, Technology) + spikes and sprint zero

ADM: capabilities, application boundaries, technology standards.

Agile: architecture spikes, thin vertical slices, “sprint 0” for risky unknowns.

Gaming examples:

Business: separate gameplay from transactions capability — buy-in cannot block the tick

Applications:REST matchmaking → assign table → WebSocket for live session.

Technology:RabbitMQ for work routing on the hot path Kafka later for history and analytics ADM chose when Agile shipped which module first.

Phase E (Opportunities) + PI / roadmap

ADM: work packages, dependencies, build vs buy vs reuse.

Agile: prioritized backlog, capacity, dependency ordering across squads.

Gaming example: offline Spark segmentation ADM framed build vs EMR buy Agile time-boxed the 10-day build proof. Architecture named the tradeoff Agile proved the increment.

Phase F (Migration) + incremental release

ADM: transition architecture, coexistence, decommission rules.

Agile: feature flags, dark launch, strangler slices, no big-bang fantasy.

Gaming example: monolith → modular monolith module by moduletransactions off the game process first, then lobby paths, then scale events. Players still at tables while the map moves underneath.

Phase G (Governance) + retro + ADRs

ADM: compliance with principles, architecture board, change control.

Agile: sprint retro, backlog refinement, lightweight Architecture Decision Records.

Gaming example:Why Hazelcast for coordination and Redisfor leaderboards?logged once CAP tradeoff named so the next squad does not reopen the same fight.

Phase H (Change management) + continuous delivery

ADM: the cycle starts again when the business changes.

Agile:continuous improvement, operational learning.

Gaming example: fraud patterns, new real-time query consumers, region/zone fleet math new ADM iteration, new Agile quarters — same marriage.

What the marriage fixed (and vs)

Without the marriage	With ADM + Agile married
TOGAF-only: 18-month target PDF; production still on monolith	ADM names split gameplay vs money; Agile ships transaction handoff first
Agile-only: six microservices, three buses, no decommission plan	REST matchmaking + WebSocket room in sequenced increments
Big-bang cutover weekend; players feel the pain	Strangler migration; live tables while map moves underneath
Same architecture fight every squad, every year	ADRs + principles — CAP, brokers, build vs buy — recorded once

Without marriage — TOGAF-only smell.

18-month target architecture PDF production still on the monolith squads ignore it.

Without marriage Agile-only smell.

Six microservices in six sprints three message buses no decommission plan matchmaking polls REST for the game loop.

With marriage:

ADM says split gameplay and money and sequence the brokers. Agile ships transaction handoff via RabbitMQ in sprint N, matchmaking REST + WebSocketin sprint N+1, Kafka for BI when the business case is proveneach increment demonstrable at a live table.

Anti-patterns we said no to.

ADM once a year, Agile ignores it — architecture becomes wallpaper

Every sprint reinvents architecture — no standards, no reuse

Phase F on a slide, cutover on a panic weekend — players feel it

We’re Agile, we don’t need stakeholders — Risk, Payments, and Support still exist on gaming platforms

We’re TOGAF, we don’t need sprints — latency and competitors do not wait

Wrapping-Up

A lot of people confuse ADM and Agile Some TOGAF practitioners ignore Agile. Some Agile warriors ignore TOGAF.

Marry both.

ADM — what to change, who cares, how to migrate, what to govern iteratively

Agile — what ships this sprint, how we learn, how we improve — iteratively

On a ive gaming platform, that marriage produced standard, beautiful results — not slide-deck perfection, but map plus motion modular engines, doors and rooms for clients, brokers where they fit, real-time paths that scale, decisions recorded so the next squad does not burn the same fuel.

ADM is not the enemy of Agile. Agile is not the enemy of ADM. Confusion is the enemy. Marriage is the practice.

September 12, 2026

TOGAF Is Theory Until You Apply ADM to a Real Problem

As a TOGAF practitioner, I hear the same complaint often TOGAF is too theoretical.

That is only half true.

TOGAF looks heavy when you study it as a framework. When you apply it the right way, especially through the Architecture Development Method (ADM), it becomes surprisingly simple and practical. Whenever a complex business problem lands on your desk, try following ADM. It does not give you magic answers it gives you a clear picture what matters, who owns it, what to build, what to govern, and what to change next.

In this article, I map a real-time fraud engine we built for high-scale gaming and payments onto ADM not as a certification exercise, but as a working method.

TOGAF ADM applied to a real-time fraud engine Preliminary through Phase H with requirements at the center signal → detect → decide → case → learn contrast of without ADM vs with ADM.

Why a real-time fraud engine is a good ADM proving ground

High-scale gaming and payments are a bad place for vague architecture.

Business risk is immediate (loss, chargebacks, regulatory heat, player trust).

Stakeholders disagree on “success” (risk wants catch-rate product wants low false positives; ops wants explainability finance wants cost).

Decisions sit on hot paths game and wallet flows cannot wait for a leisurely batch job.

Data is everywhere (events, payments, device, behavior, cases) and late or wrong data becomes wrong blocks.

Buy vs build vs reuse shows up hard (vendor fraud suites vs in-house rules/ML vs shared event platforms).

Change never stops (new abuse patterns weekly). Phase H is not optional poetry.

If ADM only works on calm greenfield HR portals, it is not Enterprise Architecture. A real-time fraud engine is the stress test.

ADM in one sentence (before we use it)

ADM is an iterative method to move from “why are we changing?” to “what must be true in business, data, applications, and technology?” to “how do we deliver and govern without lying to ourselves?”

You do not need every artifact. You need every phase question answered well enough to decide.

Preliminary Phase — get permission to architect, not just to code

ADM question: Are we allowed to run architecture as a controlled change, with principles and scope?

For a real-time fraud engine, Preliminary is where enterprises usually skip and later regret.

What we fixed here:

Architecture principles the engine must obey (examples prefer explainable decisions on player-facing blocks no silent irreversible wallet actions without audit reuse enterprise event sources before inventing parallel telemetryprotect PII in case files protect hot-path latency budgets).

Org touchpoints:Risk, Payments, Customer Support, Legal/Compliance, Game/Product, Data Platform, Security.

Repository / decision log: fraud architecture decisions are recorded, not trapped in Slack.

Scope fence: this cycle is “real-time fraud decisioning and case workflow for online play and payments” — not “boil the ocean AML + KYC + every analytics dashboard.”

Without Preliminary, every squad invents private fraud truth. ADM starts by making the fraud engine an enterprise concern.

Phase A — Architecture Vision name the pain and the target state in business language

ADM question: What problem are we solving, for whom, and what does success look like?

Fraud conversations love tools. Phase A refuses tools first.

Drivers we made explicit

Abuse and payment fraud leaking into loss and support load

Real-time game and wallet paths needing decisions in tight latency budgets

Investigators needing a case trail, not only a model score

High-scale concurrency peak play and payment bursts, not demo traffic

Management wanting to avoid unbounded third-party fraud-platform cost where an in-house + selective-buy mix is defensible

Vision statement (shape we aligned on)

A real-time fraud engine that consumes trusted enterprise events, decides with rules and models under clear SLAs on gaming and payment hot paths, opens explainable cases, and improves continuously without a second shadow data estate.

Stakeholders and concerns

Stakeholder	Concern
Risk / Fraud ops	Catch rate, case quality, tooling
Product / Game	False positives, player friction
Payments	Auth/capture risk, chargebacks
Support	Clear reasons, override path
Platform / EA	Reuse events, avoid duplicate stacks
Finance	Loss vs platform TCO

Phase A outputs a vision stakeholders can argue with. If they only argue about vendor logos, you are still in sales mode, not ADM.

Phase B — Business Architecture how fraud work actually runs

ADM question:What business capabilities, value streams, and processes must change?

For a real-time fraud engine, Business Architecture is the difference between a model demo and an operating model.

Capabilities we mapped

Signal intake- player, session, payment, device, bonus, gameplay events

Detection — rules, velocity, graph/collusion signals, ML scores

Decisioning — allow / challenge / hold / block / step-up (sync on hot paths)

Case management — queue, investigate, evidence, disposition

Feedback — confirmed fraud / false positive back into rules and training

Reporting — loss, funnel, SLA, auditor views

Value streams (simplified)

1. Live play / payment → real-time risk check → decision → continue or friction

2. Post-event review → enrichment → case → action

3. Analyst improvement → pattern found → rule/model change → governed release

Baseline vs target (business)

Baseline: fragmented checks in payment and game services tribal knowledge weak case continuity vendor tools overlapping enterprise data

Target: shared real-time fraud capability clear RACI between risk ops and engineering decision SLAs by channel feedback loop owned

If Phase B is skipped, “fraud AI” projects optimize the wrong verbs.

Phase C — Information Systems Architecture (Data + Applications)

ADM question: What data and applications realize those capabilities?

Data Architecture

Canonical risk events aligned to enterprise event models (game + payment + identity signals)

Feature / profile for real-time attributes (velocity, device reputation, linked accounts)

Decision and reason codes as first-class data (not log line archaeology)

Case and evidence store with retention and access control

Label / outcome data for model and rule learning

ADM forces a hard line: the fraud engine does not get a private parallel universe of “almost the same” player events. Reuse enterprise data products build fraud-specific semantics on top.

Application Architecture

Application Architecture building blocks: Decision API, detection workers, rules and policy, model scoring, case management, admin/simulation, and integration adapters.

Application building block	Responsibility
Fraud Gateway / Decision API	Real-time sync decisions on hot paths
Detection workers / stream jobs	Async scoring and pattern detection
Rules & policy service	Versioned, testable rules
Model scoring service	ML inference with fallback
Case Management app	Investigator UX and workflow
Admin / simulation	Shadow rules, backtests
Integration adapters	Game, payments, wallet, support, notifications

Baseline often shows rules hard-coded inside payment or game services. Target separates decisioning from channel apps so high-scale gaming and payments call one fraud engine instead of each inventing one.

Phase D — Technology Architecture make non-functionals boringly explicit

ADM question: What technology standards and patterns meet the SLAs?

For a real-time fraud engine, NFRs are the architecture:

Latency — hot-path decision budget on game and payment calls

Availability— fail-open vs fail-closed policy by transaction type (a conscious business choice)

Throughput — peak concurrent games and payment bursts

Auditability— every material decision replayable

Security— least privilege on cases; encryption; secrets

Observability — decision metrics, drift, queue age, false-positive rate

Technology patterns that fit this class of engine:

Event backbone already in the enterprise

Low-latency decision service horizontally scaled

Stream/nearline enrichment without blocking the sync path

Datastores fit to access patterns (profiles vs cases vs features)

Isolation of model runtime from rule runtime so one failure mode does not blind both

Phase D is where buy/reuse/build returns vendor case tools might be buy event bus reuse decision service build. ADM does not mandate build. It mandates fit.

Phase E — Opportunities & Solutions packages, not wishlists

ADM question: What solution building blocks and work packages close the gap?

We grouped work so delivery could breathe:

1. Foundation — decision API, reason codes, audit log, adapters (payment + one game path)

2. Detection depth — velocity rules, device/link signals, model score integration

3. Case ops — investigator workflow, evidence attach, disposition codes

4. Intelligence loop— features, backtest harness, governed rule/model release

5. Decommission— retire duplicate checks and overlapping vendor modules

Each package has dependencies and a business outcome. “Big bang fraud rewrite” is not a Phase E output. It is a resignation letter.

Phase F — Migration Planning sequence risk reduction

ADM question:In what order do we move, with what transitional architectures?

Real-time fraud migration is dual-run friendly if you design it

Shadow mode: new engine scores beside old checks compare

Dark launch: decide but do not enforce on selected cohorts

Enforce on lower-blast-radius payment types first

Expand to hotter game paths when false-positive SLAs hold

Keep break-glass override with support/risk RACI

Roadmap is calendar + risk, not only story points. ADM Phase F makes that respectable in front of finance and risk leadership.

Phase G — Implementation Governance architecture is a gate, not a spectator

ADM question: Are delivery teams implementing the architecture we agreed?

Governance checks that mattered for our engine

New channel cannot embed local fraud ifs — must call the Decision API

Reason codes required for player-facing friction

Fail-open/closed matches policy matrix

Hot-path latency budgets are measured, not assumed

PII in cases classified and retained correctly

Model/rule changes go through simulation evidence

Without G, ADM becomes a kickoff deck. With G, ADM becomes how the enterprise stays honest.

Phase H — Architecture Change Management: fraud never sits still

ADM question: How do we respond when the enemy changes tactics?

Fraud is adversarial. Phase H is continuous:

Monitoring abuse pattern shifts and model drift

Intake for new typologies from investigators

Architecture runway for new signals (new payment rail, new game type)

Re-entry to earlier ADM phases when the vision or capability map breaks

If your TOGAF practice has no Phase H operating rhythm, you certified a museum.

Requirements Management — the center that a fraud engine will flood

Every phase dumps requirements into a managed set latency, explainability, retention, jurisdictions, payment-scheme rules, player-experience limits, audit.

ADM’s center is not bureaucracy. It is how you stop a single loud incident from silently rewriting enterprise principles.

What applying ADM changed

What applying ADM changed: without ADM versus with ADM on the real-time fraud engine.

Without ADM	With ADM on the real-time fraud engine
Vendor demo drives scope	Vision and capabilities drive scope
Engineers argue tools first	Business architecture names the work
Shadow data copies appear	Data architecture reuses enterprise events
Hot path latency is a surprise	NFRs are Phase D contracts
Big-bang cutover fantasy	Migration with shadow and dark launch
Architecture finishes at design review	Governance and change management continue

TOGAF did not give us the fraud engine. ADM gave us a way to decide and sequence one for high-scale gaming and payments.

How to steal this for your next Architecture Board

1. Pick a real problem with enemies, regulators, or revenue on the line.

2. Walk A→H as questions, not as mandatory 40-deliverable cosplay.

3. Write principles in Preliminary before tool shortlists.

4. Force Business Architecture before ML heroics.

5. Make fail-open/closed latency, and explainability explicit in Technology Architecture.

6. Package E/F so risk shrinks every release.

7. Keep G/H alive or admit you only did waterfall with extra shapes.

Whenever a complex business problem lands on your desk, try following ADM. It will not hand you magic answers. It will hand you a clear picture.

Wrapping-Up

TOGAF looks heavy when you study it as a framework. It becomes simple when you apply ADM to something that can hurt the business.

For us, that something was a real-time fraud engine for high-scale gaming and payments. Stakeholders, capabilities, data, applications, technology, migration, governance, and change under adversarial load.

Theory becomes useful the day you stop redrawing the ADM crop circle and start answering its questions against a working method.

September 12, 2026

Hazelcast and Redis in Gaming

We did not pick one cache — we mapped each workload to CAP

On a live gaming platform, someone always asks: “Hazelcast or Redis?”

As if one logo must win the whole cluster.

We did not choose that way. After modularizing game engines — lobby, game, transactions on their own paths — we needed recoverable game state when a node dies, coordination across JVM services, and high-volume leaderboard reads that could tolerate a little lag. Hazelcast and Redis each fit a different CAP shape.

This article is Hazelcast and Redis — not vs. How we use a remote Hazelcast distributed cluster for replicated table state and failover, and Redis for leaderboards. CAP is the lens; live tables, pinned game nodes, and money kept off the tick are the examples.

Architecture, to us, is not picking logos. It is design, prototypes, and naming tradeoffs out loud. Choose C or A because the business can live with the downside — CAP is a lens, not a bible.

CAP in one minute

Letter	Plain meaning
C — Consistency	Every read sees the latest write (or you get an error)
A — Availability	Non-failing nodes keep answering
P — Partition tolerance	System keeps running when the network splits

P is not optional at scale. Under partition you pick C (refuse stale / fence / error) or A (keep serving, maybe stale).

If the business says…	Lean toward	Gaming example
“Wrong money or double owner is unacceptable”	C (refuse / fence / error)	Table ownership, routing on failover
“Empty screen or long outage hurts more than slightly stale data”	A (keep serving)	Leaderboard top 100, presence hints

Workload	CAP lean	Tool
Live table state + failover	C when JVM or grid member fails	Hazelcast IMap + CP
Leaderboard top 100	A — empty board worse than stale rank	Redis ZSET
Wallet / payout	Durable C	DB + queue

Same company, same platform — different maps, different clusters, different backup settings.

Our model: pinned node + IMap replica

We pin one game to one game node. That node is the active writer — pot, seats, turn, timers — single owner, local memory, fast tick.

Every state-changing action (fold, call, raise, timer) also replicates a snapshot to Hazelcast `IMap` on a remote cluster with sync backup. If the pinned node dies, a new node loads that snapshot and resumes — same pot, same turn.

Layer	Role
Pinned game node (local)	Active writer during play — every fold/call/raise mutates local state first (fast tick)
Hazelcast IMap (sync backup)	Recovery copy — replicate snapshot on each state change; new node loads table if pinned node dies
CP lock + fencing	Only one game node may resume the table after failover
Table → node routing (IMap)	Where clients reconnect after failover
Queue + DB	Buy-in / payout — durable money truth, not per-action IMap

Normal action: local mutate → IMap.set(tableId, snapshot) → broadcast

Game node dies: CP lock → new node IMap.get → hydrate → clients reconnect

oney still flows queue + DB — not per-action cache ledger.

Hazelcast: remote distributed cluster (not embedded)

Dedicated six-member grid. Game / lobby / transaction services are clients only.

Model	What it means	What we did
Embedded	Each game JVM is also a Hazelcast member	No — different scale lifecycles
Client → remote cluster	Game services are HZ clients; grid is dedicated	Yes — our model

[ Game node A — active writer ]  ──client──┐
[ Game node B — failover target ] ──client──┼──►  [ Hazelcast cluster ]
[ Lobby service ]                ──client──┘         game state IMap (sync backup)
                                                    routing IMap, CP locks

Why remote cluster: scale game fleet independently; blast radius; tune sync backup on grid nodes; game JVM churn does not reshape partitions.

Jobs we gave Hazelcast

Job	Why Hazelcast (remote cluster)
Replicated game state (IMap)	Every state-changing action → snapshot to IMap with sync backup; new game node loads table if pinned node dies
Table → node routing	Central IMap; all services know where each table lives
Ownership on failover	CP subsystem + fencing — only one node may resume the table
Soft metadata	Presence, session hints — rebuildable; async OK

CAP justification: losing a live money table mid-hand, or two nodes serving different pots, is unacceptable → sync backup on game-state `IMap`, CP + fencing on failover.

What we did not do: two writers without fencing wallet ledger in HZ async-only zero-backup on game-state maps.

Redis: leaderboards (AP-leaning)

Job	Why Redis
Global / seasonal leaderboards	ZSET — score-ordered ranks, top-N in one structure
High read volume	Simple, fast, operationally familiar
Ranking display	Product tolerates seconds of staleness across replicas

Leaderboards are AP-friendly: show rank #6 when true rank just became #5 beats an empty board. Redis ZSET, high reads, replica lag OK for display.

What we did not do: Redis as sole payout source; linearizable global #1 on every read full match simulation in Redis.

Workload	CAP preference	Choice	Why
Live table / pot / turn	Strong C per match	Game node + HZ IMap	Local active writer; replicate snapshot to sync-backed IMap for failover
Table → node routing	C on failover	Hazelcast	Sync backup on dedicated HZ members; game nodes are clients
Ownership after node death	CP (no split-brain)	Hazelcast CP	One successor; fence old owner
Buy-in / payout handoff	Durable, ordered	Queue + DB	Not cache-as-ledger; gameplay stays off money path
Leaderboard / ranking	A over strict C	Redis	ZSET top-N; brief staleness OK; high read QPS
Presence / soft hints	Eventual OK	HZ async or TTL	Rebuildable; not money truth

When things fail

Game node goes down

1. Pinned node dies mid-match

2. One successor acquires CP lock (fences old node)

3. ‘IMap.get(tableId)` → rebuild local state

4. Routing updated → players reconnect

5. Resume or void by product rules

CAP: C on failover — we paid sync replication during play.

Hazelcast grid member goes down

1. HZ member loss → backup promotes → short rebalance

2. Game clients retry `get`/`set`

3. Replicated game state survives on promoted copy

Avoid: state living only in one JVM with no recoverable `IMap` copy.

Anti-patterns

One cache for everything — CAP differs by workload
Redis Redlock for table ownership — we used Hazelcast CP + pinned node
Hazelcast for every leaderboard — Redis ZSET won for AP serving
Async / zero-backup on game-state `IMap`
Strong consistency everywhere — unnecessary for presence and ranks

Wrapping-Up

Hazelcast and Redis — not vs.

Hazelcast — remote cluster: replicated game state in `IMap`(sync backup), routing, CP on failover. Active writer on pinned node; recoverable copy on the grid.

Redis — leaderboards: available top-N, briefly stale OK.

CAP: pick C or A per workload. Architecture: design, prototype, name tradeoffs. Sync backup costs latency — we paid it where failure consistency mattered. Replica lag costs perfect ranks — we paid it where availability mattered.

Draw the workload on CAP first. Then place Hazelcast, Redis, and the game server.

if your platform debates “Hazelcast or Redis” as one winner — what workload would you map first?

September 12, 2026

REST, gRPC, GraphQL, WebSocket — When to Choose Which

What we keep missing in the Architecture Review Board — and the drawbacks of each pattern

When a new requirement lands, we open the Architecture Review Board (ARB).

The room fills quickly. Mid-level engineers come prepared — and passionate. Someone says: “We’ll do gRPC.” Someone else: “GraphQL is better for the UI.” Another: “Just expose a REST API — everyone knows it.”Occasionally WebSockets enter the chat because “we need real-time.”

I understand the concern. Everyone is trying to move fast and pick a modern, credible tool.

But sometimes we miss the fundamentals between all of them.

gRPC, GraphQL, REST, and WebSocket are not competing logos. They are different communication patterns. They answer different questions about how long the conversation lasts, who is calling, how much of the data the client needs, and what fails when the network gets ugly.

In this article I explain when to choose which communication pattern, and — just as important — what the drawbacks of each are. The examples come from high-scale gaming and payments platforms we built: matchmaking, live tables, real-time query APIs, and service-to-service paths behind the client.

Same ARB energy. Clearer criteria.

What the ARB should ask before naming a protocol

Before anyone says “gRPC” or “GraphQL,” force these questions:

1. Is this a short request/response, or a living session?

2. Is the caller a browser/app, a partner, or an internal service we control?

3. Does the client need a fixed contract, a flexible read shape, or a stream of events?

4. Are we optimizing for universality and debuggability, binary efficiency, or push latency?

5. Who owns versioning when the contract changes?

If you cannot answer those, you are not choosing a pattern. You are choosing a buzzword.

REST — when to choose it

Choose REST when the interaction is “ask and get an answer”: short-lived, resource- or command-oriented, and best served by ordinary HTTP (methods, status codes, gateways, caches).

Matchmaking was a scalable REST service. The client called it first. The service read live tables/seats from cache, applied rules, and returned game table info (later including a sticky cookie). Only then did the client know where to play.

The Dynamic Real-Time Query Engine exposed REST APIs so CRM, ML, and targeting could send a behavioral query and get a decision. Request/response. Wide consumer set. Spring Boot + HTTP was the interoperable door.

Why ARB likes REST (for good reasons)

Every platform can call it

API gateways already know how to auth, rate-limit, and route it

Easy to debug (`curl`, logs, status codes)

Natural for onboarding, config, “place me,” “get segment,” partner integrations

Drawbacks / backdrops

Chatty UIs — one screen may need many round trips

Over-fetching / under-fetching — one DTO rarely fits every client

Poor fit as the primary game loop — polling REST for table state is a smell

Versioning sprawl — `/v1` `/v2` and bloated payloads if governance is weak.

ARB line: If the conversation ends when the response returns, REST is the default until proven otherwise.

WebSocket — when to choose it

Choose WebSocket when you need a persistent, bidirectional channel: server push, client push, session as long as the user is inside an experience.

Where it fit for us

Gameplay was never REST polls. Early on, clients held a persistent socket for the life of the session — input in, broadcast out.

Later we moved to WebSockets behind an API gateway (auth, security, rate limits, routing).

Flow:

1. REST matchmaking → table info + sticky cookie

2. WebSocket through the gateway → live game path

3. Stay connected for the session; server fans out state

Historically we also separated lobby vs game connections so browsing spikes did not punish a live table.

Why ARB likes WebSocket

True real-time without fake polling

Efficient for many small messages after handshake

Matches how games, live ops, and collaborative UIs actually work

Drawbacks / backdrops

Operational hardness — sticky sessions, reconnect storms, heartbeats, backpressure

Gateway and LB behavior become architecture — not an afterthought

Horizontal scale is non-trivial — affinity, failover, “who owns this gameId?”

Abuse risk — open sockets are a DoS surface if rate limits and auth are weak

Wrong tool for one-shot CRUD — login and “get config” do not need a socket

ARB line: Use REST (or similar) to enter the room. Use WebSocket to live in the room.

gRPC — when to choose it

Choose gRPC when callers are services inside your trust boundary, you want a strict contract (Protobuf), and you care about efficiency, deadlines, and codegen. Where it fits in stacks like ours

Client-facing traffic stayed REST + WebSocket. gRPC earns its keep service-to-service:

High-QPS internal lookups (profile, features, decision helpers)

Strong typing across languages

Unary or streaming RPCs without inventing a private framing protocol

A fraud Decision API or internal RTQE neighbor might speak gRPC internally while broader consumers still see REST.

Why ARB likes gRPC

Compact binary payloads; strong performance story

Contract-first with breaking-change discipline

Deadlines, status codes, streaming built in

Excellent for polyglot microservices you own

Drawbacks / backdrops

Browsers are awkward — need grpc-web or a proxy; not “just fetch”

Ops complexity — HTTP/2 load balancing, observability, and client libraries must be mature

Less human-debuggable than JSON REST in a pinch

Overkill for simple public or partner APIs

False comfort — a `.proto` does not replace product thinking about failure modes

ARB line: gRPC for internal contracts. Do not force it to the client because it feels advanced.

GraphQL — when to choose it

Choose GraphQL when many clients need different shapes of the same domain, and REST over/under-fetching is slowing product teams — and you are willing to govern a schema.

Where it fits

BFF / app / admin / CRM consoles: screens that would otherwise need five REST calls or one monstrous DTO.

Where I push back hard

Hot game loops — use WebSocket state sync, not GraphQL as the table protocol

Simple commands — REST is clearer

Blind “GraphQL everywhere” — schema sprawl is an enterprise debt

Why ARB likes GraphQL

Clients ask for exactly the fields they need

One endpoint can serve many UI variants

Strong story for mobile and parallel frontends

Drawbacks / backdrops

N+1 and resolver cost — easy to create accidental database storms

Caching is harder than REST resource URLs

Authz must be field-aware — coarse gateway auth is not enough

Schema governance — without owners, GraphQL becomes a junk drawer

Subscriptions ≠ free real-time architecture — still need backplane thinking

ARB line: Choose GraphQL for read-shape flexibility. Do not choose it to avoid designing APIs.

Side-by-side: pattern vs backdrop

Pattern	Choose when	Main drawbacks
REST	Short request/response; broad clients; gateways	Chatty UIs; over/under-fetch; weak as live game pipe
WebSocket	Long-lived bidirectional session; server push	Sticky/reconnect complexity; harder to scale; easy to misuse for CRUD
gRPC	Internal service RPC; strict contracts; efficiency	Browser friction; ops maturity required; overkill for public CRUD
GraphQL	Many clients, many read shapes	N+1; cache/authz hardness; schema sprawl

How we composed them (what “good” looked like)

Conversation	Pattern
Matchmaking / “where do I sit?”	REST
Live gameplay / presence	WebSocket
CRM / ML segment query	REST (+ JSON body)
Service-to-service, high QPS, strict	gRPC (internal)
Diverse admin / app reads	GraphQL (when UI diversity demands it)
Durable async side effects (e.g. money path)	Message broker — different layer, not a fifth “API style”

What I say in the ARB when the suggestions fly

When someone says “we’ll do gRPC / GraphQL / REST,” I translate:

REST — “We need a door everyone can knock on.”

WebSocket — “We need a room that stays open and pushes.”

gRPC — “Two services we own need a tight, fast contract.”

GraphQL — “Many UIs need many shapes of one graph — and we will govern the schema.”

If the sentence is only “it’s modern,” that is not an architecture decision.

Wrapping-Up

In the ARB, mid-level energy is valuable. Protocol fashion is not.

Sometimes we miss the funda: REST, gRPC, GraphQL, and WebSocket solve different communication problems, and each brings drawbacks you must budget for — chattiness, connection ops, browser proxies, schema and resolver risk.

When a new requirement opens the board, don’t start with the acronym. Start with the conversation type. Then choose the pattern. Then name the backdrops out loud so nobody is surprised in production.

Enter with REST when you need a door.

Stay with WebSocket when you need a room.

Speak gRPC when services need a tight contract.

Offer GraphQL when many UIs need many shapes — and you will own the schema.

September 11, 2026

RabbitMQ and Kafka

Use each where it fits — utilize the features, don’t crown a winner

In 2016, we redesigned our game engines.

We were moving from a classic monolith to a modular monolith — lobby, game, transactions, and related concerns split by responsibility so each piece could scale and fail on its own. Once those modules stopped living in one process, they needed a way to talk without blocking each other on synchronous calls.

That seam was a message broker. We chose RabbitMQ.

This article is not RabbitMQ vs Kafka. It is RabbitMQ and Kafka — how we used each where it fit, utilized the features each one is actually good at, and stopped treating messaging as a single-tool religion.

The problem the redesign created

In the monolith, “send this to transactions” was often just a function call in the same process. After the split, a buy-in or a payout could not sit on the critical path of live gameplay. The game server had to publish and keep moving. The transaction server had to consume and act when it was ready.

We needed:

1. Point-to-point communication — some messages had a clear owner. One producer, one consumer path. Work had to land with the right module, not spray across the system by accident.

2. Broadcast when every consumer must act — some messages were not “pick one worker.” Every interested consumer had to see the message and do its own work.

3. Work distribution under load — for other flows, we needed many consumers sharing the load in round-robin style so no single worker became the bottleneck.

4. Survival under failure — if a consumer crashed mid-flight, the message could not vanish. It had to stay at the broker until something healthy consumed it.

5. High availability at the broker layer — if one broker node went down, the platform still had to serve. Gaming traffic does not wait for a maintenance window.

6. Serious throughput — we were designing for the order of 1 lakh (100,000) concurrent messages being consumed. This was not a toy queue for nightly jobs.

RabbitMQ mapped cleanly onto all of that.

Why RabbitMQ fit that architecture

Point-to-point where ownership was clear

For flows like game server → transaction server, we needed point-to-point messaging. Publish a command or event meant for one consumer path. Decouple the producer from the consumer’s speed. Keep gameplay off the money path.

RabbitMQ made that natural: publish to an exchange, bind a queue to the module that owns the work, consume with acknowledgements. The producer does not care whether the consumer is momentarily slow. The broker buffers.

Topic when every consumer must act

Not every message was “give this job to one place.”

Some messages needed every interested consumer to consume and act. For those, we used topic exchanges. Each consumer (or each consumer type) bound its own queue with the routing pattern it cared about. One publish, many independent reactions.

That gave us fan-out of intent without hard-coding a list of callers inside the producer. New consumers could subscribe by binding — the publisher stayed dumb about who was listening.

Fanout for queue round-robin under load

Where the job was pure work distribution — many workers, same kind of work, share the load — we used fanout into queues that consumers competed on in round-robin fashion.

One message → one worker. Add more consumers, get more parallelism. That pattern is RabbitMQ at its most boring and most useful: a durable work queue in front of a pool of processors.

Between topic (everyone who cares acts) and fanout / competing consumers (one worker acts), we covered both “notify the system” and “do the job once.”

Durable queues: messages stay until consumed

This was non-negotiable in a gaming stack that touched money and live state.

We used durable queues so that:

Queues survived broker restarts.

Messages were not treated as ephemeral fire-and-forget.

If a consumer broke, the message stayed at the broker until a healthy consumer took it and acknowledged it.

That durability model matched how we thought about reliability: the broker is the safety net between modules. A crash in one modular service must not erase in-flight work for another.

In practice that meant designing for at-least-once delivery. Consumers had to be idempotent. Duplicates can happen when a consumer dies after doing the work but before the ack. Durable queues protect you from loss. They do not invent exactly-once magic. We accepted that trade-off consciously.

Dead letter queues for transactional outages

Durable queues keep messages alive. Dead letter queues (DLQs) gave us a place to put work that could not complete cleanly.

When a transaction path hit an outage — consumer errors, rejected messages, retries exhausted, or a downstream money path temporarily unhealthy — those messages did not disappear into a black hole. They landed on a dead letter queue, where we could inspect them, retry them, or run a controlled recovery once the outage cleared.

For gaming, that mattered as much as the happy path. Buy-ins and payouts cannot be “lost because the consumer threw.” DLQs turned transactional failure into something operable: quarantine, diagnose, execute again when the system was ready.

So the reliability story had two layers:

Durable queues — message stays until a healthy consumer acknowledges it.

Dead letter queues — failed or unprocessable transactional work stays available for recovery after outages.

Absolute match: act in milliseconds, not replay history

For our game engines, RabbitMQ was not “good enough.” It was an absolute match for what a broker had to do.

Gameplay and money-adjacent flows had to act within milliseconds. The job of the broker was to get the right message to the right consumer fast — point-to-point, topic, or round-robin — and keep that path reliable under failure. We did not need to save a long history of every message so someone could replay last week. Once the work was done (or parked on a DLQ for recovery), the broker had finished its job.

That distinction is easy to miss when you evaluate messaging tools in the abstract. Game engines live in the present tense. Publish, route, consume, act. RabbitMQ is built around that shape of work, and in production it worked fine for us at the load and availability bar we needed.

High availability: keep serving when a node dies

Durability at the queue level was only half the story. The other half was broker high availability.

We ran RabbitMQ as a distributed cluster. The requirement was simple and absolute: if one node went down, the system should keep serving. Producers and consumers had to continue — buy-ins, payouts, and cross-module work could not freeze because a single broker machine failed.

That is where RabbitMQ’s distributed mechanisms mattered for us:

Queues and messages were set up so work survived node loss, not only process crashes on the consumer side.

Clients could reconnect and resume against healthy nodes.

The cluster absorbed a node failure without turning the modular game engines back into a tightly coupled outage.

Distributed brokers also force you to think about network partitions and split-brain — the ugly cases where nodes cannot agree on who is in charge. For our gaming requirements, we operated the cluster so those distributed behaviors stayed within what the platform could tolerate: failover worked, serving continued, and we did not see the broker layer become the weak link when a node disappeared.

In a live gaming environment, “high availability” is not a slide. It is whether tables keep moving and money paths keep accepting work at peak. RabbitMQ’s HA model met that bar for us.

Scale: 1 lakh concurrent messages consumed

People sometimes dismiss RabbitMQ as “fine for small systems.” That was not our experience.

Configured and operated carefully — durable queues, dead letter queues, HA across nodes, sensible prefetch, enough consumers, monitoring on queue depth and consumer lag — RabbitMQ handled the load we needed: on the order of 100,000 concurrent messages being consumed across the paths that mattered for the modular game engines.

We did not pick it because a blog said it was trendy. We picked it because our patterns were route and act in milliseconds, with availability under node failure and **DLQs for transactional outages — and the broker could take the concurrency we were aiming at.

Then the business grew: analytics and BI

As the business grew, analytics and BI arrived with a new class of requirements.

They needed game events and transactional events stored in the BI system — routinely — so the business could analyze what players did and how money moved. Data lakes and warehouses entered the conversation. Snowflake and similar systems became the place where transformed events were supposed to live for reporting and analysis.

Our first instinct was natural, and a little dangerous: we already had RabbitMQ in the middle of the game engines. So we suggested RabbitMQ again.

The plan was simple on paper:

1. BI / analytics consumers bind and consume from the relevant exchanges.

2. They transform the messages.

3. They store the results into the BI system / data lake (for example Snowflake).

For some days, that also worked fine. Events flowed. Transforms ran. Rows landed in the warehouse. It looked like we had extended the same broker to a new audience without inventing new infrastructure.

That comfort did not last.

The day risk analysis changed the question

One fine day, the risk analysis team came with a harder ask.

They did not only want a live feed into BI. They wanted to transform events in ways the BI team could modify — evolving logic, re-shaping pipelines, changing how raw game and transactional events became analytical facts. And they needed the underlying messages available for reconciliation the next week.

That sentence is where the architecture cracked.

RabbitMQ had been perfect when the job was: deliver this work now, act in milliseconds, then you are done.It is a weak fit when the job becomes: keep the events around, let multiple teams re-transform them, and let risk come back next week to reconcile against what actually happened.

We had stretched a work broker into an event history problem.

Once a message is consumed from a classic queue path, it is not sitting there as a week-long source of truth for reconciliation. DLQs help with failed processing. They are not a BI-grade archive of every game and transactional event for next week’s risk checks. Asking every new team to “just consume from the exchange” also couples analytical evolution to the same live routing fabric the game engines depend on — and it does not give risk a clean way to re-read last week’s facts after BI changes a transform.

That is when it became clear: for this case, we had chosen the wrong broker.

Not wrong for game engines. Wrong for analytics + BI + risk reconciliation over time.

The requirements had shifted from act now to retain, re-transform, and reconcile later. Same events. Different job. Different tool.

Introducing Kafka for BI and analytics

So we introduced Kafka for the BI / analytics path.

Not as a replacement for RabbitMQ on the game engines. As the right backbone for a different job: game events and transactional events that must be stored, analyzed as routine, re-transformed when BI logic changes, and still available when risk comes back next week for reconciliation.

Kafka for BI and analytics: game and transactional events plus Cassandra CDC feed a commit-log cluster; transform consumers reshape events in transit; BI, risk reconciliation, and other analytics read independently into the data lake.

What Kafka gave us that RabbitMQ was not designed to be for this case:

A commit log, not a disposable queue

Kafka is built as a distributed commit log. Events are appended to topics and retained by policy. Consumers track an offset — a position in history — instead of “take this message and it is gone from the shared truth.”

That is the mental model analytics needed. The stream is a durable record of what happened. Risk can return to last week’s events. BI can change a transform and re-read. The log stays.

History you can replay

For reconciliation and evolving analytics, history is the feature.

With Kafka, game and transactional events remain available for the retention window we set. A consumer that was wrong on Monday can be fixed on Thursday and replay. A risk job that needs last week’s facts does not depend on someone having saved a side copy “just in case.” The platform keeps the events.

That is exactly what broke when we tried to stretch RabbitMQ: once the live consume path had moved on, next week’s reconciliation had no first-class history to stand on.

Transformation in transit

BI and risk did not only need raw events parked forever. They needed message transformation in transit — shape the event on the way into lakes and warehouses (Snowflake and similar), and keep the ability to change that shaping as analytical definitions evolved.

Kafka fits that pipeline shape:

1. Producers (or bridges from the game / transaction world) publish canonical events to topics.

2. Stream / transform consumers read the log, apply BI-owned logic, and write curated results downstream.

3. When the transform changes, teams can redeploy the transform and, where needed, replay from history instead of begging for a one-off re-extract.

4. Risk can consume the same underlying topics independently for reconciliation — without stealing messages from the BI consumer or coupling to the game-engine RabbitMQ fabric.

The important architectural point: **transformation becomes a consumer concern on top of a retained log**, not a one-shot side effect of emptying a work queue.

CDC from Cassandra into Kafka

Kafka also fit another reality of our stack: we already had important state in Cassandra.

For analytics and BI, it was not enough to stream only the messages the game engines happened to publish. We needed changes from Cassandra tables themselves — inserts and updates that represented durable game and transactional facts available as an event feed.

So we integrated Cassandra → Kafka as CDC (change data capture). Table changes were captured and published into Kafka topics. From there, the same commit-log strengths applied: retain history, transform in transit, land curated data in the BI lake, and let risk reconcile later by re-reading the stream.

That pattern worked well. CDC turns the database into a producer of facts without forcing every service to remember to emit a perfect analytics event on every write. Kafka is a natural sink for that feed because it is built to hold an ordered, replayable log of changes — not to empty a work queue and forget.

Use each where it fits

After Kafka landed for BI and analytics, we were not declaring a winner. We were **using each technology where it fit** and utilizing the features that matched the job:

We did not “upgrade from RabbitMQ to Kafka.” We utilized RabbitMQ for work routing and Kafka for the commit-log / history path. Both stayed. Both earned their place.

Where it fit	What we utilized
Game engines: route work, act in milliseconds	RabbitMQ — exchanges, durable queues, DLQs, HA
BI / analytics / risk: history, transform, reconcile later	Kafka — commit log, replay, transform in transit
Cassandra table changes into the analytical stream	Kafka CDC

What I’d tell a team building the same platform

Do not start with “which technology is better.” Start with the job, then utilize the features that fit:

1. Point-to-point? You need clear ownership of work — RabbitMQ exchanges and queues fit well.

2. Every consumer must act? Topic-style fan-out to many queues.

3. Round-robin workers? Competing consumers on shared work queues.

4. Consumer can die? Durable queues and ack-based consumption so messages stay until processed.

5. Transaction path can fail?Dead letter queues so outages become recoverable work, not silent loss.

6. Broker node can die? A real HA cluster — including partition / split-brain behavior, not only the happy path.

7. Act in milliseconds, no long history? A classic broker like RabbitMQ is often the cleaner fit — that was our game-engine case.

8. Analytics, BI, or risk need events next week — to re-transform or reconcile? Utilize a commit log with history — for us, that was Kafka, including CDC from Cassandra.

We asked the first set of questions in 2016. RabbitMQ fit. We utilized it.

We asked the later questions when the business grew. Kafka fit. We utilized it.

That is the whole lesson.

Wrapping up

Do not compare technologies to pick a champion. Use them wherever they fit. Utilize their features.

RabbitMQ fit our modular game engines: point-to-point, topic broadcast, fanout with round-robin, durable queues, dead letter queues, HA, and roughly 1 lakh concurrent messages consumed— act within milliseconds.

Kafka fit BI, analytics, and risk: a commit log, history for next-week reconciliation, transformation in transit into lakes like Snowflake, and CDC from Cassandra tables.

That is why this article is named RabbitMQ and Kafka — not RabbitMQ vs Kafka.

Same company. Same events. Different jobs. Both tools, used on purpose.

September 10, 2026