Serving Qwen3.5-0.8B with a 0.016% weight edit: measure the loops, pin the sampling on the server

A model that never stops talking is a capacity problem first. Each answer that runs to the token limit (tokens are the word pieces a model reads and writes) holds a slot on your GPU until the limit cuts it off, and leaves nothing to parse.
Qwen3.5-0.8B, a small open model from the Qwen team, writes a long reasoning scratchpad before it answers, and its model card warns that this size is "more prone to entering thinking loops". Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL published a fix with no training. A large model they call GPT-6 Astra read the small model's internals and changed 64 columns of weights (the learned numbers) in each of two layers: 131,072 weights, 0.016% of the model. The Hugging Face user akhaliq, working with ML Intern on HF Jobs, rebuilt and verified the file and reran MATH-500, a standard set of 500 competition math problems.
We put both files on one RTX 4090 on our cloud, each served by vLLM (an open-source engine that turns a model file into an HTTP API), on the first 50 MATH-500 problems. The tables are in our side-by-side test of the edited and base Qwen3.5-0.8B. Here is what to get right to repeat it.
Track how many answers hit the limit, not only accuracy
A score hides where the GPU time went. For a model that loops, watch the share of answers the limit had to cut.
| 50 MATH-500 problems, 32,768-token limit | Edited | Base |
|---|---|---|
| Right answers | 62% | 8% |
| Cut at the limit | 20% | 96% |
| Average tokens per answer | 20,444 | 31,832 |
For the same 50 problems, the edited model wrote 1.02 million tokens and the base 1.59 million. Per right answer, about 33,000 against 398,000, a rough ratio since only 4 base answers were right. Our 62% also beats the published runs (48.1% from the authors over all 500, 51.0% from akhaliq). On our 50 problems akhaliq's run scored 54%, so we read the margin as sample size.
The edit takes away an obstacle, not a gap in knowledge. On Gemma 3 270M, where only 8.5% of answers hit the limit, the same method left MATH-500 between 6.8% and 7.7%.
The trap: sampling settings your API does not forward
Sampling settings decide how the model picks each next token. The authors' figures assume thinking on, temperature 0.6, top-p 0.95, top-k 20 and a presence penalty of 1.5 (it discourages repeating tokens already used). Change one and you measure something else.
Per request, our platform's API passes on temperature, top-p and presence penalty. It drops top-k and the chat template switch that enables thinking. So we set them all on the server:
--default-chat-template-kwargs '{"enable_thinking": true}' \ --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "presence_penalty": 1.5}' vLLM prints the override at startup. Check that line before trusting a benchmark. A parameter your gateway silently drops looks exactly like a weaker model. The complete serve command and the reason each flag sits on the server are in the engineering appendix of the Qwen3.5-0.8B weight edit write-up.
Two engines on one card do not split it evenly
Each engine got 0.45 of the 24 GB card, which peaked at 16.9 GiB. But engines that boot together check free memory at the same instant. From that identical setting, the base engine ended up with 6.48 GiB of KV cache (the memory that holds each running conversation) and the edited one with 3.32 GiB. The edited one queued requests sooner, so its throughput is not comparable. Boot the engines one after another.
A streamed answer can end without telling you
Through our endpoint, one base answer simply stopped at 152 seconds. The client got no error and no end marker. We suspect the platform's two-minute silence timeout, whose error chunk our example script never reads. It is not confirmed, and the issue is open. Whatever stack you use, discard any streamed answer that never sends a finish_reason (the field that says why generation ended), and check each chunk for an error key.
Sizing: caps on a big card, not small cards
We capped the 4090's memory and reran five solved problems on the edited model alone. A 6 GB cap returned the same five right answers as 8 GB. At 5 GB, vLLM refused to boot until we lowered --max-num-seqs (how many requests it serves at once) from 256 to 32. At 4 GB it never started. No real small card was tested, and real ones have slower memory, so expect slower runs.
What it does not cover
We tested math with thinking on. Chat, instruction following, thinking off and images are untested. We also skipped the cheaper base-model fix Qwen's card suggests: a lower limit, or stopping a loop from the stream. The AI Search video mentions about 44% for the unedited model with other settings, with no source named. We have not checked it.
Credit goes to the Qwen team for the model, and to the MIT CSAIL authors for the method, the diff and the script that rebuilds and hash-verifies the edit. akhaliq built the checkpoint and ran the independent rerun. The vLLM project wrote the engine, and the AI Search channel is where we spotted the story. We served both models on one card, ran the 50-problem comparison and tested the memory caps.
The setup, the full memory ladder, the timings and an SDK example that runs both models and deletes them on exit are in Qwen3.5-0.8B: an AI edited 0.016% of its weights, and thinking loops fell from 93% to about 32%.