What missing ADRs cost when you build a Dynamic Real-Time Query Engine — not Slack archaeology
When a new requirement lands, we open the Architecture Review Board (ARB).
Someone says “buy another CRM tool.” Someone says batch is fine. Someone says “we already decided this.” Nobody can find where.
That is not a tooling problem. It is a memory problem — and on a live gaming platform, forgotten decisions become production incidents, re-opened fights, and sprints that undo last year’s work.
Architecture Decision Records (ADRs) are how you stop paying twice. Not bureaucracy. Insurance.
This article is not “what is an ADR” from a textbook. It is a consequence map: decisions we made on a real gaming platform — Dynamic Real-Time Query Engine for CRM and ML, lobby and live tables, wallets, caches, fraud on the hot path — and what broke when we did not record them (or recorded them in Slack and lost them).
ADR in one sentence (before the horror stories)
An ADR is a short, durable note: what we decided, why, what we rejected, and what we will revisit. It lives in the repo or architecture wiki — not in a thread that scrolls away.
You do not need one ADR per line of code. You need one per decision that will be fought again if nobody wrote it down.
The decisions that deserved an ADR (gaming platform)
On our platform, these were not “nice to have.” They were forks in the road:
| Decision | Why it mattered |
|---|---|
| RTQE as platform, not CRM patch (Cassandra + JSON DSL + shared API) | Millisecond behavioral queries for CRM and ML — one live truth |
| JSON DSL vs open SQL on RTQE | CRM self-serve segments without unbounded queries |
| REST for lobby, WebSocket for live play | Door vs room — different conversation shapes |
| Hazelcast for table state, Redis for leaderboards | CAP — coordination vs eventually-fresh reads |
| One writer per ledger | No duplicate wallet paths during strangler migration |
| Fail-open vs fail-closed on fraud | Block play vs allow play when the engine is down |
| Reuse Spark / buy EMR vs build segmentation | Cost and ownership on offline compute |
| Strangler order: transactions before lobby split | Plateau sequence — not big-bang weekend |
Each of these showed up in ARB, stand-up, or a new squad’s first design doc. When we missed recording one, we paid a predictable price.
If you miss the ADR — consequences (with examples)
1. Dynamic Real-Time Query Engine — “let’s bolt another tool onto CRM”
Decision: Build the Dynamic Real-Time Query Engine (RTQE) as shared platform infrastructure — not a one-team patch. Cassandra time-series (one table per event type), JSON DSL for behavioral segments, REST APIs for CRM, targeting, and the ML Service — millisecond queries while the player is still on screen.
If you skip the ADR:
CRM buys or builds a narrow segmentation tool; ML spins up a second live-events pipeline.
Consequence: duplicate stores, segments that lag live play, engineering tickets for every new behavioral question — “real-time” becomes tomorrow’s batch.
Symptom in prod: offer fires after the player left the table; CRM and ML disagree on the same player’s last hour.
What the ADR should have said: Platform not patch. One engine, many consumers. Bounded JSON DSL — not open SQL. Cassandra model: player_id partition, time-series clustering, one table per event type.
2. RTQE query model — “just expose SQL to CRM”
Decision: JSON DSL with bounded query models (aggregates, time windows, expressions) — every query maps to a single-partition, time-bounded Cassandra read. Not open SQL to marketing tools.
If you skip the ADR:
Someone exposes read-only SQL “for flexibility.”
Consequence: unbounded scans, hot partitions, on-call pages during a campaign launch — CRM “self-serve” becomes production roulette.
Symptom in prod: segment job takes 30 seconds; player already churned; CRM blames “the real-time engine.”
What the ADR should have said: DSL not SQL. Query bounds by construction. `searchTime` aligns with time-series clustering key.
# ADR-001: Dynamic Real-Time Query Engine as shared platform (not CRM patch)
Status: Accepted
Date: 02/15/2025
Deciders: Architecture Board, CRM, Data Platform, ML, VP Engineering
Context
CRM and marketing needed to understand live player behavior and act while the player is still on screen not via nightly batch jobs. Existing CRM tools could not answer behavioral questions in milliseconds. The easy path was another point solution on the CRM stack; the harder path was shared platform infrastructure.
Decision
Build the Dynamic Real-Time Query Engine (RTQE) as a foundational building block:
Apache Cassandra — one table per event type, time-series keyed by `player_id` + event timestamp
JSON DSL — bounded behavioral queries (aggregates, time windows, expressions) — not open SQL
Event ingest pipeline — decoupled producers into the engine (point-to-point queues)
Java 21 + Spring Boot — REST APIs for CRM, targeting, and ML Service consumers
Platform scope — same engine serves multiple consumers — not a single-team patch
Consequences
Positive: Millisecond segments on live play; CRM self-serve in JSON; ML gets real-time features from one pipeline; ~100K req/s headroom with virtual threads and horizontal scale.
Negative: Custom DSL to maintain; Cassandra modeling discipline required; platform team owns SLAs for all consumers.
Obligations: Query bounds by construction; one-table-per-event-type schema governance; document rejected “SQL over everything” and “CRM-only tool” paths.
| Alternative | Why rejected |
|-------------|--------------|
| Bolt-on CRM segmentation tool | Solves one team; duplicate pipelines for ML; batch mindset |
| Expose raw SQL to CRM | Unbounded queries; safety and ops risk on hot path |
| Single generic events table | Partition and query unpredictability; fights Cassandra strengths |
| Nightly batch only | Misses the conversion window — player already left |
3. REST lobby + WebSocket game — “let’s poll matchmaking from the socket”
Decision: REST (or HTTP API) to Lobby — find and join a table. WebSocket to Game — live play, broadcast state.
If you skip the ADR:
Mobile team polls lobby state over the game socket every 500ms for simplicity.
Consequence: socket fan-out load explodes; matchmaking logic bleeds into game shards; you cannot scale browse traffic separately from seated players.
Symptom in prod: lobby spike during a tournament drags down live tables — the failure domain you split on purpose.
What the ADR should have said:Door = REST. Room = WebSocket. Do not merge because one client library is easier.
# ADR-002: REST for lobby (door), WebSocket for game (room)
Status: Accepted
Date: Migration Plateau 2
Deciders: Mobile, Lobby, Game squads, ARB
Context
Matchmaking is request/response: find table, seat player, return connection info. Live play is a long-lived session: broadcast state, play events, timers. Mixing both on one channel couples browse traffic to seated-player scale.
## Decision
REST (HTTP API) to Lobby Service for matchmaking, presence, seat assignment.
- WebSocket to Game Service for live table play after seat confirmed.
- Session handoff: lobby returns game shard / token; client opens socket to game fleet.
## Consequences
Positive: Scale lobby on browse traffic independently; clear failure domains; matches client mental model (door vs room).
Negative: Two client integrations; handoff bugs if token/session contract drifts.
Obligations: Document handoff flow in API spec; integration tests for seat → connect path.
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| WebSocket for lobby polling | Fan-out and unnecessary persistent connections for short requests |
| REST for live play | Polling latency unacceptable for table state; wrong conversation shape |
| Single monolith socket | Failure domain and scale coupling we were leaving |
4. Hazelcast vs Redis — the annual CAP debate
Decision: Hazelcast (remote cluster, sync backup, CP locks) for recoverable table state when a game node dies. Redis (cluster) for leaderboards where slightly stale ranks are acceptable.
If you skip the ADR:
Every new hire proposes Redis for everything or Hazelcast for “leaderboards too.”
Consequence: quarters lost to benchmarks and slide decks; wrong tool on wrong workload; wallet or pot state in a cache that was never meant to be the ledger.
Symptom in prod: failover restores the wrong pot; leaderboard shows ranks from before a partition.
What the ADR should have said:HZ = coordination + recovery (lean C). Redis = high-read scores (lean A). Money still flows queue + DB.
# ADR-003: Hazelcast for table recovery, Redis for leaderboards
Status: Accepted
Date: Post modularization scale-up
Deciders: Platform ARB, Game squad
Context
Game nodes are pinned per table with local active state. On node death, we must recover pot, seats, and turn without split-brain. Leaderboards are high-read, tolerate slight staleness, and do not need CP semantics for every rank update.
Decision
Remote Hazelcast cluster (six members, clients only from game/lobby services): `IMap` with sync backup for table snapshots; CP locks + fencing on failover.
Redis cluster (three nodes) for leaderboards reads/writes — eventual consistency acceptable for ranks.
Wallet ledger stays queue + database — not in either cache.
Consequences
Positive: Correct recovery for live money tables; leaderboard scale without HZ partition pressure on every rank tick.
Negative: Two cache platforms; clear ownership required per workload.
Obligations: Failover drills; monitor backup lag; no dual writers on game state.
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| Redis for table state | Wrong CAP fit for pot recovery without careful fencing; risk of split brain |
| Hazelcast for leaderboards | Overkill; CP cost not justified for ranks |
| Embedded HZ in game JVM | Blast radius; fleet churn reshapes grid |
5. One writer per ledger — “we’ll dual-write during migration”
Decision: During strangler migration, only Transaction Service writes the wallet ledger. Monolith wallet path must decommission, not linger “just in case.”
If you skip the ADR:
Plateau 1 ships with two writers — monolith and new service both mutate balance.
Consequence: race conditions, support tickets (“where did my chips go?”), reconciliation nightmares, audit failure.
Symptom in prod: player balance disagrees with ledger after a hand — the gap ArchiMate was supposed to close.
What the ADR should have said:One writer per ledger. Coexistence duration named. Decommission rule in the same doc as the migration ADR.
ADR-004: One writer per wallet ledger during strangler migration
Status: Accepted
Date: Plateau 1 (transactions strangler)
Deciders: Transactions squad, Platform ARB, Risk
Context
Extracting Transaction Service while the monolith still serves lobby and game creates a coexistence window. Dual wallet paths are tempting for “safety” but produce race conditions and audit gaps.
Decision
Transaction Service is the sole writer to the wallet ledger after cutover.
Monolith in-process wallet mutation
decommissioned by named date — not left dormant. Coexistence: read paths may dual-serve briefly; writes do not.
Consequences
Positive: Auditable ledger; support can trust one source of truth; strangler gap closes cleanly.
Negative: Requires feature flags and migration testing; rollback plan must be explicit.
Obligations: ADR linked in ArchiMate gap P0→P1; reconciliation job until decommission complete.
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| Dual-write monolith + service | Race conditions; “where did my chips go?” tickets |
| Big-bang cutover weekend | Unacceptable downtime risk for live tables |
| Ledger in game cache | Violates money-off-hot-path principle |
6. Fail-open vs fail-closed — fraud on the hot path
Decision: On player-facing blocks, prefer explainable decisions and clear audit. On hot-path latency, define what happens when the fraud engine is unavailable — fail – open vs fail-closed is a business choice, not a default in the SDK.
If you skip the ADR:
On-call flips behavior under pressure without a recorded principle.
Consequence: either legitimate players frozen at peak (revenue loss) or abuse runs unchecked during an outage (loss + trust hit).
Symptom in prod: incident post-mortem says “we thought prod was fail-closed” — staging was fail-open.
What the ADR should have said:Gameplay path: [X]. Wallet path: [Y]. Owner: Risk + Platform. Review when SLA changes.
# ADR-005: Fail-open vs fail-closed on the fraud hot path
Status: Accepted
Date: Fraud engine Phase D (Technology Architecture)
Deciders: Risk, Payments, Platform ARB, Game squad
Context
The real-time fraud engine sits on game and payment hot paths. When the Decision API is slow or unavailable, someone must decide: block traffic (fail-closed) or allow traffic (fail-open) until recovery. That is not an SDK default — it is a business and compliance choice. Without a recorded policy, staging, prod, and on-call behave differently.
Decision
Define a policy matrix by path and transaction type — documented, tested, and owned by Risk + Platform:
| Path | Default when engine unavailable | Rationale |
|------|----------------------------------|-----------|
| Gameplay / join table | Fail-open with rate limits + post-hoc review | Revenue and player experience at peak; friction on outage freezes live tables |
| Wallet / buy-in / payout | Fail-closed (or queue + async settle with hold) | Money movement requires audit; abuse during outage is unacceptable |
| High-value withdrawal | Fail-closed always | Chargeback and regulatory exposure |
Cross-cutting rules:
Explainable reason codes required for any player-facing block when engine is healthy
Shadow mode in migration — score beside legacy; do not enforce until false-positive SLA holds
Break-glass override with Risk RACI — not an undocumented on-call toggle
Same matrix in staging and prod — no silent drift
Consequences
Positive: Incidents have a playbook; no “we thought prod was fail-closed”; ARB can govern Phase G against a written matrix.
Negative: Fail-open on gameplay accepts temporary abuse risk — must pair with velocity caps, DLQ review, and post-incident replay.
Obligations: Latency budgets measured; policy matrix in runbooks; ADR linked from Decision API config and ArchiMate technology constraints.
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| One global fail-closed | Legitimate players frozen at peak; revenue loss during engine blip |
| One global fail-open | Wallet abuse during outage; audit and trust failure |
| On-call decides per incident | Inconsistent behavior; post-mortem blame without policy |
| Hard-code in payment service only | Game path undocumented; six squads, six truths |
| Vendor SDK default | Business policy hidden in library; not governed by ARB |7. Build vs buy vs reuse — “let’s build our own Spark”
Decision: Reuse Apache Spark for offline segmentation; buy managed EMR where the bill fit; build only the thin control plane (scheduling, tenancy, guardrails) — not a second distributed compute engine.
If you skip the ADR:
A squad starts a 10-month “lightweight batch engine” because EMR “felt expensive” on one slide.
Consequence: duplicate platform, no hiring market, security review from scratch — savings illusion.
Symptom in prod: two batch systems, neither owned; CRM segments diverge from real-time truth.
What the ADR should have said:Reuse Spark. Buy EMR when TCO wins. Build only orchestration we cannot buy.
# ADR-006: Build vs buy vs reuse for offline segmentation (Spark)
Status: Accepted
Date: Phase E (Opportunities & Solutions) — after RTQE platform
Deciders: Platform ARB, Data Platform, CRM, Finance
Context
CRM needed offline behavioral segments that aligned with the Dynamic Real-Time Query Engine — same player truth, batch scale for campaigns and analytics. The commodity capability is distributed batch compute. The enterprise question was not “Spark or not Spark” but reuse the engine, buy managed EMR, or build a custom runtime.
Decision
Compose — do not pick one label for everything:
| Choice | What we chose | What it means |
|--------|---------------|---------------|
| Reuse | Apache Spark | Offline segmentation engine- commodity distributed compute; do not reinvent |
| Buy (evaluated) | Amazon EMR — not selected for this workload | EMR scales; for our utilization pattern the bill was ~10x a lean in-house runtime |
| Build | Thin control plane only | Spark master on EC2 (published IP); worker pods autoscale; custom scaler (5‑min warm-up, min/max per job); ~10 days to deliver |
Rules recorded:
Reuse what is commodity (Spark). Buy when TCO and operating model both fit. Build only the layer that turns reuse into enterprise fit — not a second batch engine.
Offline segments must trace to the same event semantics as RTQE — no shadow data estate for CRM.
Owner: Data Platform operates runtime, scaler, and upgrades.
Consequences
Positive: ~10× lower ongoing cost vs EMR path for our pattern; delivery in ~10 days; CRM segments stay aligned with real-time truth.
Negative: We own Spark master failover, scaler logic, and security patches — not EMR’s managed boundary.
Obligations: Revisit EMR when utilization or compliance changes; document payback in ARB; no “build our own Spark” proposals without new context.
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| Buy EMR for this workload | Capability fit yes; cost fit no (~10× bill for our utilization) |
| Build a new batch engine from scratch | Enterprise malpractice; no hiring market; years of maintenance |
| CRM vendor segmentation module only | Duplicate pipeline; diverges from RTQE player truth |
| Reuse Spark with no control plane | Could not meet warm-up, per-job limits, tenancy we needed |
| Buy only — no reuse/build split | Treated “buy” as slogan; ignored thin-build payback |
Strangler order — “let’s split lobby first, it’s easier”
Decision: Transactions module first via broker — money off the monolith hot path. Lobby + game split in Plateau 2. Sequence documented in migration model and ADRs.
If you skip the ADR:
Product pushes lobby microservice first because the UI team is ready.
Consequence: pretty services, money still in the monolith — the riskiest coupling untouched; false sense of “we migrated.”
Symptom in prod: “we’re on microservices” but a stuck wallet call still kills live tables.
What the ADR should have said:Plateau 1 = transactions strangler. Plateau 2 = REST lobby + WebSocket game. Link to ArchiMate gap.
ADR-007: Strangler migration order — transactions before lobby split
Status: Accepted
Date: ADM Phase F (Migration Planning)
Deciders: Platform ARB, Product, Game, Transactions, Lobby squads
Context
Moving from a socket monolith (lobby + game + wallet in one deployable) to modular services tempts teams to split what is easiest first — often lobby/UI — while money stays in the monolith. That creates a false "we migrated” story and leaves the highest-risk coupling (gameplay tick + wallet) untouched. Migration order is an architecture decision, not a sprint convenience.
Decision
Strangler sequence — documented in ArchiMate plateaus and ADRs:
| Plateau | Work package | What ships | Monolith still owns |
|---------|--------------|------------|---------------------|
| Plateau 0 | — | Baseline monolith | Lobby, game, wallet (all in one) |
| Plateau 1 | WP1 — Extract transactions | Broker + Transaction Service; one ledger writer | Lobby + game |
| Plateau 2 | WP2 — Split lobby and game | REST lobby + WebSocket game handoff | — (target modular) |
| Later | WP3+ (optional) | Shard game fleet, containerize, autoscale | Per roadmap |
Rules recorded:
1. Plateau 1 = transactions strangler first — money off the gameplay hot path before cosmetic service splits.
2. Plateau 2 = REST lobby + WebSocket game — only after Plateau 1 coexistence rules and ledger ADR (ADR-004) are met.
3. No big-bang weekend — each plateau runs in production with named gaps and decommission dates.
4. Product priority does not override sequence without new ARB context (supersede this ADR).
Consequences
Positive: Riskiest coupling addressed first; live tables keep running; board can recognize plateau transitions; aligns with ADM Phase F and Agile increments.
Negative: Lobby/UI teams wait for Plateau 2; requires patience and visible migration model so squads do not fork private diagrams.
Obligations: ArchiMate work packages link to ADR-004 (one writer) and this ADR; sprint reviews ask “which plateau does this close?”
Alternatives considered
| Alternative | Why rejected |
|-------------|--------------|
| Lobby microservice first (UI ready) | Money still in monolith; stuck wallet call still kills live tables |
| Big-bang cutover | Unacceptable downtime; no coexistence learning |
| Game split before transactions | Money path remains on hottest failure domain |
| Target-state-only migration (skip Plateau 1) | Poster architecture; production never matches model |
| Parallel all services at once | Six squads, three buses, no decommission plan |
The pattern when ADRs are missing
| What you feel | What is actually happening |
|---|---|
| “We already decided this” | Decision lived in Slack / a meeting / one senior’s head |
| “Why did they build it that way?” | No rejected-options trail — looks like incompetence, was often a tradeoff |
| “New squad, same fight” | Organizational amnesia — cheaper to write 1 page than rerun ARB |
| “Rollback panic” | Nobody remembers coexistence rules from migration |
| “Architecture is slow” | You are re-deciding instead of referencing |
ADRs do not slow you down. They buy back the quarters you lose re-fighting “batch CRM vs real-time platform” — or Hazelcast vs Redis.
What to put in an ADR (minimum viable)
1. Title — e.g. `ADR-001: Dynamic Real-Time Query Engine as shared platform (not CRM patch)`
2. Status — Proposed | Accepted | Superseded
3. Context — pressure, constraints, plateau you are in
4. Decision — one clear paragraph
5. Consequences — positive, negative, what we owe ops/security
6. Alternatives considered — what you said no to and why
Length: one to two screens. If it reads like a thesis, nobody will write the next one.
Where it lives: `docs/adr/` in the repo, or wiki linked from the ArchiMate work package and the Jira epic. Same decision, three doors — one truth.
ADRs × ADM × Agile × ArchiMate
| Practice | Role |
|---|---|
| ADM Phase G | Governance — decisions are controlled, not oral tradition |
| Agile | ADR per meaningful increment — not per sprint theater |
| ArchiMate | Work packages link to ADR IDs — gaps close with named decisions |
| ARB | Accept or reject ADRs; do not re-debate without new context |
Anti-patterns (ADR edition)
Slack as ADR — searchable until it is not; context dies on scroll
ADR after the outage — post-mortem theater, not architecture
ADR per README paragraph — team stops writing them
No “superseded”— two contradictory ADRs, both “accepted”
ADR with no rejected options— future you cannot defend the call
Architecture PDF instead of ADRs — pretty, disconnected from backlog
Checklist: does this decision need an ADR?
1. Will a new squad ask “why?” in six months?
2. Did we reject something credible?
3. Does it affect money, play, or compliance on the hot path?
4. Does it sequence migration (plateau, coexistence, decommission)?
5. Will ops need to know failover behavior?
If yes to two or more — write the ADR before merge, not after the incident.
Wrapping up
On a live gaming platform, every architecture decision you skip recording, you pay for twice: once when you make it, again when someone unknowingly unmakes it.
RTQE as platform. REST vs WebSocket. Hazelcast vs Redis. One writer per ledger. Fail-open vs fail-closed. Build vs buy vs reuse. Strangler order.
None of those are secret wisdom. They are decisions — and decisions without ADRs become mysteries, then incidents, then quarters in ARB.
Record the call. Name what you rejected. Link it to the plateau.
That is EA doing its job: memory over slogans.
Note on the sample ADRs in this article (ADR-001 through ADR-007 — RTQE platform, REST/WebSocket, Hazelcast/Redis, one ledger writer, fail-open/fail-closed, build vs buy vs reuse, strangler order): they are illustrative mocks — written in real ADR shape to show context, decision, rejected options, and consequences. They are not the literal records from our Architecture Review Board. In production, your ADRs live in your repo or wiki, carry your IDs, dates, deciders, and may differ in detail — use the samples as templates, not as copies of our internal governance pack.
