As an Enterprise Architect, I collaborate with business stakeholders and engineering stakeholders all the time. Outcomes, SLAs, capacity, latency, failure modes — that rhythm is familiar.
Data scientists who build the ML algorithms were a different experience in a good way.
They showed up with business flavour and tech flavour in the same conversation. In one breath simulation accuracy, what churn means for the player, payment and bonus buckets. In the next, XGBoost, Random Forest, sklearn self-learning paths, training windows on ten years of history. They were not pure business and not pure platform engineering. They sat between and that made the Architecture Review Board (ARB) interesting.
| Stakeholder | What they brought | What the ARB had to decide |
| Business | Payment/bonus buckets, retention goals, act in session | Differentiating or operational? What is the SLA? |
| Data science | 80% offline accuracy, XGBoost/RF/sklearn, feature definitions | What ships in v1? Frozen artifact vs offline retrain? |
| Engineering | Event-driven engine, Flask/FastAPI, RTQE, concurrency | Serve at burst? Who owns the seam? |
At the ARB we had to translate three worlds into one production decision:
Business — act while the player is still in session churn capacity and purchase capacity must map to real buckets
Data science — 80% simulation accuracy on churn and purchase models demo landed; artifacts ready
Engineering — can we execute those models in real time when game actions end, at gaming concurrency, without the prediction arriving after the player left?
If you are not married with your ML — ML aligned with engineering — you will not get results. Not “ML vs engineering” Not a handoff and goodbye. A marriage analytics owns the models platform owns real-time execution both meet at the seam where game events become predictions and buckets.
That is where the marriage happens not in a hallway after the demo, but in the ARB room when all three flavours are in the same frame. If you run an ARB, the useful question for ML is not only “what is the accuracy?” It is: “What must be true. in production for this model to count as a result?”
Our ML analytics team had done serious work. They built churn and purchase models on ten years of player history, ran simulations, hit around 80% accuracy, demoed the room applauded. Then the ask landed on architecture when a game action ends in real time, execute the right model, return a prediction, and recommend payment and bonus buckets according to churn capacity and purchase capacity.
That is a different problem from the notebook. An accurate model with no event-driven platform, no serving layer, and no infra under concurrent load does not change outcomes. It stays in a slide deck.
This article is how we built that platform a Python Prediction Engine on Flask first, upgraded to FastAPI when concurrent predictions became the bottleneck and what we learned when ML and engineering had to work as one system.
Accuracy is necessary. It is not sufficient.
| What ML delivered | What the platform had to deliver |
| Models on 10 years of history | Real-time scoring on live game events |
| 80% simulation accuracy | Low latency under production concurrency |
| Demo on sample data | Integration with game engine and targeting |
| Model artifacts handoff | Routing by event — churn vs purchase handlers |
| Works in the lab | Payment & bonus bucket recommendations in session |
The gap between those columns is architecture and engineering not another week of feature engineering alone.
That gap closes only when ML and engineering are married, not when one team “finishes” and throws artifacts over the wall the same lesson we learned at the ARB before a line of platform code shipped.
The business problem: act on the game, not on yesterday’s warehouse
Players do not churn in batch files. They churn in sessions.
When a game ends, we already know something useful: how they played, whether they won, how they bet, how long they stayed. The business wanted to use that moment to
Estimate churn risk and steer retention offers
Estimate purchase capacity and place the player in the right payment / bonus bucket
Do it in real time— not after a nightly Spark job.
So the game engine would emit an event whenever a relevant game action completed. Our job was to consume that event, call the right model, and return a recommendation the targeting and payments stack could act on.
That became the Prediction Engine.
How we designed it: event-driven, command pattern, handler factory
We designed an event-driven flow.
Game engine (game ends / action completes)
→ event on the bus
→ Prediction Engine consumes
→ HandlerFactory picks handler by event type
→ Model inference (churn or purchase)
→ Response: bucket / offer recommendation
→ CRM / targeting / payments path
Command pattern + HandlerFactory
Not every event should run every model. A game-end event might need a churn handler. A purchase-intent or cash-action event might need a purchase-capacity handler. We used the Command pattern.
Each event type maps to a handler(a command object)
A HandlerFactory returns the right handler from the event name / type
The handler loads features, calls the model, shapes the response
That kept the engine extensible. New model, new event — add a handler and register it in the factory. The game engine stayed dumb emit event, do not know model internals.
Flask v1: ship fast, prove the path
We built the first version in Python with Flask.
Simple REST endpoints for health and internal ops
Consumers pulled work from the messaging layer and invoked handlers
Gunicorn workers behind a load balancer
Containerized, load-tested, concurrency tests passed, deployed
For a while it worked fine. Predictions landed. Buckets updated. The demo became production.
What broke with Flask under high concurrent predictions.
Load tests passing once is not the same as sustained concurrent predictions at gaming traffic.
Flask’s default synchronous WSGI model became the ceiling
| Symptom | Under the hood (Flask sync WSGI) |
|---|---|
| Latency spikes at peak | Each request blocks a worker for full inference path |
| Queue depth grows | More game-end events than free Gunicorn workers |
| Upstream timeouts | Targeting / CRM cannot wait past SLA |
| “Add workers” stops helping | CPU-bound predict() does not scale linearly with threads |
| GIL pressure | Python threads contend on CPU during heavy inference |
We were not failing because the models were wrong. We were failing because the serving model — one blocking request per worker — could not accommodate the concurrency profile once many tables ended games at once.
Eighty percent accuracy does not help if the prediction arrives after the player left the screen.
Why we upgraded to FastAPI (and what it actually fixed)
This is not Flask vs FastAPI as a religion. It is match the framework to the load.
We moved the Prediction Engine to FastAPI because we needed a better concurrency story at the API and orchestration layer.
| Layer | FastAPI benefit for Prediction Engine |
|---|---|
| I/O-bound work | Async routes while waiting on features, cache, downstream calls |
| Contracts | Pydantic — fewer bad payloads hitting handlers |
| Integration | OpenAPI — CRM and internal consumers onboard faster |
| Inference | CPU-bound predict() in thread pool — do not block event loop |
| Operations | Async shell + bounded inference workers — clearer scaling model |
FastAPI did not make the churn model smarter. It made the platform able to serve many concurrent predictions without worker exhaustion on the hot path.
What we kept from the Flask design.
Event-driven ingestion — unchanged idea
HandlerFactory+ Command handlers — unchanged pattern
Containerized deployment and autoscaling — still required
Models from the analytics team — same artifacts, better house.
The upgrade was the serving and concurrency layer, not a rewrite of the ML story.
Thread pools at consumers — and why events still piled up
Under concurrent load we used thread pools at the message consumers. That was reasonable for v1 each game-end event got a worker thread, the handler ran, features were fetched, XGBoost / RF / sklearn`predict()` executed, the result went out.
It worked until events piled up.
When many tables ended games at once, the pattern looked like this.
Event 1 → thread blocked (feature fetch + predict)
Event 2 → waits for free thread
Event 3 → waits …
Queue depth grows → latency grows → buckets arrive too late.
Python’s thread model was part of the pain:
A bounded thread pool caps throughput — backlog is inevitable under burst
Many threads mean memory and context-switch overhead, not free parallelism
Blocking I/O (RTQE, Cassandra, HTTP) ties up threads while they wait
CPU-bound `predict()` keeps threads busy for the full inference window
We were not GPU-bound. We were concurrency-shaped wrong for a burst of single-row tabular scoring.
| Layer | Thread pool at consumer (v1) | Async + Uvicorn / uvloop (v2) |
|---|---|---|
| Consume event | One OS thread per in-flight event | await — many events multiplexed on event loop |
| Feature fetch (RTQE) | Thread blocked while waiting on network/DB | await — thread not held during I/O wait |
| XGBoost / RF predict() | Thread blocked for full CPU inference | run_in_executor — bounded pool, off event loop |
| Under burst (game ends) | Pool exhausted → events pile up → latency spikes | I/O stays cheap; inference capped but queue drains faster |
| GPU needed? | No — tabular CPU models | No — fix serving shape, not hardware |
The async pattern: Uvicorn, event loop, and bounded inference
The fix was not “more threads.” It was async for waiting, thread pool for predicting.
We moved the Prediction Engine to FastAPI served by Uvicorn, with uvloop as the event loop where we could — a faster asyncio loop for I/O-heavy work. The pattern.
async consumer / handler
├─ await consume event (I/O — event loop handles many in flight)
├─ await fetch features (I/O — RTQE / cache / HTTP)
├─ await run_in_executor(predict) (CPU — bounded thread pool, NOT on event loop)
└─ await ack / publish result (I/O)
Rules we enforced.
1. Never call `predict()` directly inside `async def`— it blocks the event loop and recreates the pile-up.
2. Bounded inference pool — cap concurrent XGBoost/RF/sklearn calls (semaphore or fixed executor size).
3. Async for orchestration — consume, fetch, ack let the loop multiplex I/O.
4. Scale out — more container replicas when queue depth rises async is not infinite capacity.
uvloop did not make the models more accurate. It made the consumer path stop drowning in thread overhead when events stacked up. GPU would not have fixed that either our models are tabular, CPU-native, one player per game end.
This is the same lesson as on the Java side with virtual threads at RTQE scale: match the concurrency model to what actually blocks I/O on the loop, CPU in a pool.
The full stack picture
Our models were tabular — not deep learning. The analytics team trained on ten years of history using XGBoost, Random Forest, and sklearn self-learning / incremental models. Simulation accuracy was strong (~80%). Serving them in production was a different shape of problem:
| Model family | Role in Prediction Engine | Serving note |
|---|---|---|
| XGBoost | Primary churn and purchase scorers | CPU inference — one player per game-end event; GPU not required |
| Random Forest | Supporting / ensemble paths (sklearn) | CPU-only; keep forests lean for millisecond budgets |
| sklearn self-learning | Incremental updates offline; frozen artifact in prod | Retrain on schedule — do not partial_fit on every live event |
GPU does not improve accuracy for this stack. It can help large-batch deep learning — not single-row XGBoost/RF scoring at game end. The bottleneck we hit was concurrent serving(Flask workers), not matrix math on a GPU.
Flow in production.
1. Game engine— live table ends; emits event (game end, cash action)
2. Message bus— decouples gameplay from scoring (same event-driven seam as the modular game engines).
3. Prediction Engine (FastAPI)— consumes event; HandlerFactory picks churn vs purchase handler.
4. Feature fetch— live behavioral features from RTQE and related real-time services (often more latency than `predict()`).
5. Model inference (CPU) — handler calls the right XGBoost / RF / sklearn artifact in a thread pool.
6. Response — churn capacity → retention bucket; purchase capacity → payment / bonus bucket.
7. Targeting / CRM / payments — act while the player is still in session
Features came from the same real-time data path we had already built for segmentation the Dynamic Real-Time Query Engine and related platform services. Accurate models plus live features plus an engine that could keep up that is when recommendations started to matter in production.
Lessons we took to the architecture board
1. Demo accuracy ≠ production value. Simulation at 80% is a starting line. Without real-time integration and infra, it is a handoff to nowhere.
2. Marry ML with engineering — or you will not get results. Separate ownership is fine; separate reality is not. Analytics trained on ten years of history architecture built the event-driven engine, feature path, and serving layer. Neither side wins alone. Handoff without partnership is applause without outcomes.
3. The ARB is where ML, business, and engineering marry.Data scientists bring business and tech flavour together — simulation accuracy and model mechanics. Use the board to ask what must be true in production for the model to count as a result.
4. Join model team and platform team at the seam. Analytics builds models; architecture builds the event-driven engine that runs them on live game actions — together, not in sequence with a six-month gap.
5. Design for events, not cron. Game-end is a natural trigger. Command + factory keeps new models from polluting the game engine.
6. Load test the serving path, not just the model. Concurrency tests on Flask passed — until production traffic shape changed. Test concurrent predictions, not one request at a time.
7. Upgrade frameworks when the bottleneck is serving, not when a blog says so. Flask was the right v1. FastAPI was the right v2 for our concurrency profile.
8. Async does not fix CPU-bound inference. Run `predict()` in a thread pool or dedicated workers. FastAPI wins on orchestration and I/O, not on magic faster math.
9. XGBoost / RF / sklearn are CPU-native. Do not reach for GPU to fix accuracy or single-row game-end latency. Optimize the serving layer and feature path first.
10. Thread pools at consumers are not enough under burst. Use async for I/O(Uvicorn / uvloop) and a bounded pool for `predict()`— never block the event loop with inference.
When Flask is still fine
Internal tools, low QPS, batch scoring
Prototypes and first vertical slice
Teams that will never see game-end burst concurrency
When many game actions end at once and payments and bonus buckets must update in session, plan for serving architecture early not after the ML demo.
Wrapping up
Our ML team did serious work: ten years of history, churn and purchase models, 80% simulation accuracy, a convincing demo. None of that produces player outcomes until architecture gives you a Prediction Engine that.
Consumes game events in real time.
Routes through a HandlerFactory and command handlers.
Runs models under concurrent load
Returns payment and bonus bucket recommendations while it still matters.
We started on Flask — containerized, load-tested, deployed, working. We moved to FastAPI when high concurrent predictions exposed the limits of synchronous serving.
Marry ML with engineering. Eighty percent accuracy in simulation is not a result. An event-driven Prediction Engine at gaming concurrency is. Without that marriage models plus platform plus infra you do not get outcomes. You get applause in a meeting room.
If you have shipped ML to production — what broke first the model, the features, or the serving layer? Drop a comment.
Leave a Reply