A 125B model at 50+ tok/s on a 12 GB gaming card: what I changed in Strata
I run Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on one RTX 3060 with 12 GB of video memory and 30 GB of RAM. Nothing leaves my PC. This post is about the fork I made to get there, what I measured, and what I have not tested.
Starting point: Strata
The model is about 78 GB of files at the quantization I use (IQ3_S). It does not fit on a 12 GB card. Strata by Niko1221 and contributors (MIT) solves that by keeping the most-used experts on the graphics card, putting more in RAM, and streaming the rest from the SSD. My fork, Strata-HotRod, makes that path faster and checks that answers stay correct.
What I measured
All of this is on one machine: RTX 3060 12 GB, Intel i5-12400F, 30 GB RAM, a 500 GB NVMe on a PCIe 3.0 link, Rocky Linux 9.6.
| Answer (131K context) | tokens/s |
|---|---|
| Code, 500-650 tokens | 52-55 |
| Reasoning, about 1,000 tokens | 66-71 |
| Prose, 100-150 tokens | 50-53 |
Long prompts, 262K context: an 8,554-token prompt is read in 16.9 s and answered at 52-54 tokens/s. A 45,398-token prompt takes 67-68 s to read, then the answer comes at 38-41 tokens/s. The next turn of that same conversation starts in 0.4-0.5 s, because the saved state is reused. The first message of a chat is the slow part: it waits on the SSD for the model's experts.
What I changed
- A learned look-ahead. The engine guesses which experts the next layers will need and loads them early. Against the plain router, the guesses are right 85-88% of the time one layer ahead (router: 77-79%), 76-79% two layers ahead (66-69%), and 66-71% four layers ahead (55-58%).
- Three tiers: card, RAM, SSD. The hottest experts sit on the card, the next ones in a RAM tier sized from free memory, the rest on the SSD. Budgets and reserves are measured on your PC and kept across restarts.
- Expert placement from routing traces, instead of a fixed guess.
- Stand-in experts with a re-check. To stay fast, some of a token's experts are replaced with similar ones already on the card.
- Pinned RAM segments (on by default): about 9% faster on this PC.
Every change and its measurement is in the repo's docs, including ideas that did not help. For example, a learned read-ahead of lower-ranked guesses made decoding 23% slower on this PC, because the drive was already at its limit, so it stays opt-in.
Is the answer quality still good?
In a small test (50 maths questions, 30 coding questions, answers read by hand), the stand-ins gave 47/50 and 30/30 correct, against 48/50 and 30/30 with nothing replaced. A test this small cannot show a difference of one or two answers either way. On MMLU-Pro (15 questions from each of its 14 subjects, 210 in all, thinking on, temperature 0) it scored 87.1%, 183 of 210. That is a small sample, so read it as a sanity check, not a leaderboard result.
What I have not tested
One card, one model, one PC, Linux only. Other GPUs, AMD, Windows, macOS and other model sizes may work in upstream Strata, but I have not run them, so I will not claim them.
Try it, and tell me what you get
The repo has the install steps and the full change log: github.com/sangharshadhyeta/Strata-HotRod. If you run it on different hardware, I would like your numbers, good or bad. The repo has a short guide for how to report them.