Q4_0 vs Q5 on a 27B: prompt speed is basically a wash

2026-09-27 · Local AI · Experiments · RTX 3070 8GB, Tesla P100 16GB

Illustration of two nearly identical race cars side by side — Illustration: AI-generated locally with FLUX.2 klein.

Prompt speed from production logs (3K+ token prompts) (tok/s)

Q4_0110–170

Q595–143

The ranges overlap heavily, so Q5 costs little prompt speed on this setup.

MODEL: Qwen3.8-27B Q5
HARDWARE: RTX 3070 8GB + Tesla P100 16GB
CONTEXT: 111K
RESULT: 95–143 tok/s prompt processing on 3K+ token prompts
WHAT CHANGED: Q5 instead of Q4_0 (110–170 tok/s)

After Part 2 I had a theory: Q5 felt fine in chat, but it must be quietly costing me prompt speed compared to Q4_0. A quick test seemed to confirm it: 74–88 tok/s. That’s a big gap.

Then I went back to the real numbers.

Test runs vs. production logs

That 74–88 figure came from a couple of one-off test runs. So I pulled all 50 production server logs from actually using the Q5 model day to day and read the prompt-processing speed off every request with a prompt of 3K tokens or more:

Quant	Prompt tok/s (real use)
Q4_0	110–170
Q5	95–143

Some individual Q5 requests: a 42K-token prompt at 136 tok/s, an 83K-token prompt at 94 tok/s. Speed drops as the prompt gets longer, which is normal.

The ranges overlap heavily. On this hardware, Q5 does not cost much prompt speed at all. The quality upgrade is close to free.

Why the test lied

A few runs is a tiny sample; server logs cover weeks of real prompts.
Test prompts were short-ish; production prompts ranged up to 80K+ tokens.
Warm-up and caching behave differently in a fresh test than in a server that’s been running.

The caveat I can’t get around

Every production Q5 run used the exact same layout: tensor split 6.5 / 16 (the 3070 holds about 27% of the weights), 111K context, ubatch 128. I never ran Q5 any other way, so I can’t tell you whether a different split would help it more than Q4_0.

And remember Part 2: ubatch 128 is small. Both quants are probably leaving prompt speed on the table.

Takeaways

Don’t trust a quick benchmark over your own logs. If your server logs timings, they’re the better dataset.
On a 3070 + P100 split, Q5 vs Q4_0 is a quality decision, not a speed decision.
Next experiment: Q5 at ubatch 1024–2048 with 8K-token prompts, shifting the split toward the P100 to make room.