Dwarf Star Gate
and the Genie

The problem shows up the second you own more than one machine that runs a model.

Say you have a DGX Spark and a Mac, each running DS4, Antirez’s excellent local inference engine. That’s great — until you notice the little machine is sitting idle and the big one is melting, and you don’t know why. A request lands anywhere. Your follow-up turn lands somewhere else, loses what it had in cache, and re-reads the whole prompt like it’s meeting you for the first time. And there’s no single place to look at any of it.

None of this is a hard problem. It’s just bookkeeping: who lives where, what’s busy, what’s healthy, what kept the last few turns warm. The trouble is that it’s bookkeeping you have to do by hand, every day, while you’re trying to do something actually interesting.

So I built Dwarf Star Gate — a small local gateway that sits in front of your DS4 servers and never gets bored of the bookkeeping.

The one-sentence version

Dwarf Star Gate is a thin routing and observation layer around DS4. It does not run the models, process prompts, generate tokens, or manage KV state — the engine does all the heavy lifting, and credit for that belongs to Antirez’s DS4 project, not to me. What DSG adds is the part that makes a pile of servers behave like a fleet instead of a pile.

The two things it does first are the ones you feel — and then there’s the Genie, who tries to clean up your messes and do a bit of the admin for you.

• Durable session affinity. A conversation gets a home, and later turns go back to the same worker, so their KV cache keeps getting reused instead of thrown away. Busy conversations queue at home instead of bouncing all over the lab.
• Load-aware placement of new work. At most one active request goes through the gateway per server; the rest wait in bounded FIFO queues. It won’t convince a busy worker to take on a third thing it can’t handle.
• The Genie. DSG also ships a local assistant who watches the fleet and quietly handles some of the admin. He tries to keep your sessions afloat without dumb errors while the backend wobbles — and when something does go wrong, he tries to clean up the mess safely, with guarded, verified actions instead of grabbing the keys. He’s moving fast, so I’ll spell out exactly how he earns that below.

DSG won’t move caches, guarantee hits, manage model containers, or know how much GPU memory a server has left. DS4 owns cache validity and GPU concurrency. The gate is the concierge, not the kitchen.

The control room

Then there’s the part you didn’t know you needed: a terminal-flavoured dashboard that shows what the fleet is actually doing. Per worker you get decode tokens per second, prefill speed, prefix reuse, genuinely cold starts, resident cache misses and disk restores, plus recent request outcomes and timings.

The important habit here is that DSG would rather say unknown than invent a number. If a worker is idle, it shows the last measurement with its age rather than pretending the machine is still at full speed. A missing token counter stays missing. Positive cached tokens don’t get dressed up as proof the model is resident in RAM. That’s the boring discipline that keeps a hobby lab from fooling itself.

Workers that fail generation get quarantined until real recovery is verified — a model that still answers /v1/models while its generator is broken doesn’t get to stay in the pool. There’s draining, health probes, and an operator control surface that’s private and loopback-only.

Not a customer-facing gateway

The thing that makes DSG different from the load balancers a commercial service runs is the stakes. A gateway in front of a public API is built for strangers: isolation, per-tenant fairness, rate limits, and the comfortable assumption that a dropped request is fine because somebody just retries. Failing one request costs almost nothing.

A local hobby lab is the opposite. The work here is long-lived — a generation that runs for minutes, an agent that carries its own context across many turns, a big prompt you would rather not feed in twice. If a session falls over because of a silly error — a transient hiccup, a misread health check, a worker that answered a model-list probe while quietly breaking its generator — that is not a dropped request, that is lost work. Progress gone. Context gone. You start over, and you usually do not know why.

So DSG optimises for keeping sessions alive. Conversations stay at home, so the cache stays warm. A worker is only quarantined on real generation failure, not a transient blip. Recovery has to be verified — the worker must actually produce a generation and continue a cold-to-warm session before it is trusted again. Nothing is ever blindly replayed over a half-finished stream, because appending a different answer to a broken conversation is not recovery, it is corruption. And the dashboard would rather tell you unknown than pretend everything is fine.

Two pieces make that concrete. The first is patient waiting: when a server is unavailable or paused, DSG holds the original request and keeps the connection alive instead of failing fast with no_healthy_workers and letting your client burn through a short retry budget. It sends HTTP 102 frames so the socket stays honest while it waits, and it never replays a body it has already held.

The second is the Continuity Door — the stable client endpoint DSG puts in front of its own replaceable core, so even a planned gateway restart stops being an event. When the core has to come down, the door holds new request streams unread, lets the responses already streaming play out, swaps in a freshly probed core, and only then forwards each held call exactly once. It never spools your prompt to disk and it never replays dispatched work. The honest boundary matters too: this is not a durable message queue and it is not transparent mid-stream failover. If a worker genuinely crashes mid-generation, DSG says so — explicitly, and with its DSG Report: prefix — rather than pretending it can stitch a broken conversation back together.

In other words: for external traffic, “drop it and move on” is a feature. For the work you are doing on your own machines, “keep it alive over silly errors” is the feature. That is the different approach local AI needs, and it is the part I would defend first if anyone tried to call DSG just a load balancer.

The Genie

This is the fun part. The Genie is an optional local LLM that watches the fleet on your behalf.

It reads a compact metrics briefing — the same numbers the dashboard shows. It does not read your conversations, your prompts, or your answers. It produces observations and, when warranted, a short recommendation, printed on a color-coded health wire next to the capacity panel. Green is good, grey is info, amber is a warning, coral is critical.

The Genie doesn’t guess a diagnosis out of thin air and it doesn’t get to pretend something happened. Advice carries the timestamp of the evidence it was based on, and it gets withheld after ten minutes, or when membership, health, pause, quarantine, context or draining state changes, or when the gateway’s status is missing. Stale advice is not served. A long thinking phase alone is not a stall. A busy queue alone is not critical.

And critically: the Genie has no shell, no arbitrary routing, no model-setting, and no cache-editing tools. What it can do, when you’ve separately enrolled a service and enabled recovery, is request one bounded, guarded action — restart a dead DS4 service — through a runner that independently checks the worker is still the failed instance, that no one else owns the recovery, that a retry budget exists, and then performs a real generation and a cold-to-warm continuation check before letting the worker back in. Reinstatement requires actual success, not just a new PID.

So the Genie is an observer with one careful finger, not a janitor with a key to everything. It is advice, and it is held to a very high bar before it is allowed to ask for anything.

The XGBoost models (the part that isn’t doing anything yet)

Underneath all this, deep in the roadmap, there’s a prediction subsystem built around XGBoost. Right now it isn’t doing anything to the fleet — no routing decision depends on it. What it’s actually doing is buildingcalibration data: versioned features captured off the request path, so that one day we can train a model to help load-balance. That’s the entire point of the collection.

The features are boring on purpose — recent output and duration trends, variability, hardware class, accelerator and RAM, cache evidence, and optionally locally-generated embeddings of a bounded recent-conversation slice. A future model would predict how long a piece of work is going to take, and use that to decide where new work lands. Three distinct forecasts, separately: total service time at admission, an updated total after the upload is seen, and remaining busy time for an active request.

Here’s where the design gets opinionated. The model is trained off the request path with fixed CPU, time and data budgets. Every candidate is an immutable bundle carrying hashes of its source, schema and data. Cross-validation uses forward-time folds with purging, so a label that hasn’t finished yet is never used to pick trees. And the in-gateway tree evaluation has to match the Python XGBoost runtime numerically, to the float — not approximately.

None of it controls anything by default. A fitted model is experimental until it passes a fixed backtest and an independent future shadow gate. Prediction-assisted placement of a new session is separately armed and requires unseen-session evidence. Sparse data can legitimately mean no model ships today, and collecting more evidence does not mean relaxing the tests.

There’s a particular line in the project docs I like a lot: “No model is called better merely because it fitted, reloaded or produced an attractive chart.” The Genie can request a bounded training run or a rollback, but it can’t change its own acceptance gates. The validator, not the LLM, decides.

And here’s where it’s going. Once a model has earned its place — passed the backtest and the future shadow gate, on real evidence — the Genie owns it. He can request a bounded training run, tune the trees, and offer a rollback, all while the fixed validator holds the door. He never gets to loosen his own acceptance gates, and he never gets to point a model at live routing before it’s proven. Load-balancing becomes something the fleet does from evidence rather than guesswork — but only once the evidence says it can.

That’s the whole reason I trust this. The moment you let the thing that makes the prediction also be the thing that judges its own prediction, you’ve built a very confident liar. DSG refuses to do that.

Why this makes a hobby life easier

The honest answer is that it removes a category of boring, constant attention. You stop babysitting. Conversations stay warm, the dashboard tells you where things are slow, and when a machine dies the fleet tells you — and, with permission, quietly fixes the one case it safely can.

It’s also honest about what it isn’t. There is no live XGB routing implied, no automatic cache migration, no “learning router” marketing. The roadmap is explicit about the order of work: diagnose the reliability incidents first, then explain idle capacity, then validate embeddings, then productionise the Genie and operator UI, then deterministic recovery. The moonshot — moving a conversation to an idle compatible server while carrying its disk cache — is worth investigating precisely because it is not done yet, and because cross-device cache portability is not verified.

What I like is the shape of it. There’s a gate, which is honest bookkeeping. There’s a black hole behind it, which is the real engine doing the real work. And there’s a Genie, which is a model that’s allowed to observe and to politely ask, but is structurally incapable of running the place.

The timing (and why it’s a real need)

If that still sounds like a niche obsession, the timing just made the argument for me. Today — the same day this is going up — NVIDIA shipped PAIR, the Personal AI Router, at IFA 2026. It’s free and open source. It auto-discovers the Macs, PCs and DGX Sparks on your network and puts them behind a single endpoint, dispatching agentic sub-tasks to whichever machine has spare capacity. It isn’t a new inference engine — Ollama or LM Studio still runs the model — it’s a router. The fact that the biggest GPU company on earth decided a router for a home fleet was worth a launch is confirmation that this is a real need, not a hobby tic.

But it’s also the clearest illustration of why DSG leans the way it does. PAIR is built for swarms: lots of short, independent sub-tasks fired at whatever machine is free, where dropping one is cheap because another daemon just retries. That’s throughput, and it’s great. DSG is aimed at the other end of the same problem — the long-lived session that can’t afford to fall over, where keeping a conversation warm and alive matters more than spreading it across the house. Both are local AI; they’re just optimising for different failure models. Keeping your work afloat over silly errors is the one nobody else was building, so I built it.

The dev process is part of the point

The way DSG is built enforces the same honesty the software enforces.

• No package installation for the gateway or dashboard — no database, no Kubernetes, no frontend build system, no CDN, no analytics service, no cloud telemetry.
• One checkout. Source, your private config, and your runtime artifacts live together, with the private parts ignored and mode-0600.
• A repo-local pre-commit hook checks the exact staged blobs for secrets, private artifact names, and deployment-diary phrases — and it refuses to overwrite a custom hook setup.
• Tests run against local HTTP fixtures, not GPUs, plus privacy tests and, for the optional predictor, a locked Python environment. CI covers Linux and macOS.
• A publication policy separates reusable public guidance from private fleet diaries — your hostnames, inventories, incident logs and real session examples stay out of the public repo.

There’s even a lovely honesty note at the bottom of the README: public visibility is not a license. There is no open-source license grant yet. That’s a decision for the maintainer, and the project says so instead of pretending otherwise.

Credits

• Salvatore “Antirez” Sanfilippo and the contributors — DS4, the engine that does the actual work. The inference, kernels, quantization and cache engine are theirs; please go star the original project.
• Me — the gateway, the dashboard, the Genie, the roadmap, the stubbornness about not letting a model grade its own homework.

Code, docs, roadmap and the whole shebang: github.com/JordiPosthumus/dwarf-star-gate.