Why I wrote my own inference engine for two P100s
- MODEL
- Qwen3.8-27B, custom Q4-class mix
- HARDWARE
- Tesla P100 16GB + Tesla P100 16GB
- CONTEXT
- short prompt
- RESULT
- 49 tok/s plain decode, ~118 tok/s with MTP (3 drafts)
- WHAT CHANGED
- own engine vs patched llama.cpp (32 tok/s)
- MODEL
- Qwen3.8-27B, custom Q4-class mix
- HARDWARE
- Tesla P100 16GB + Tesla P100 16GB
- CONTEXT
- 2048-token prompt
- RESULT
- 558 tok/s prompt processing
- WHAT CHANGED
- own engine vs patched llama.cpp (~405 tok/s)
Short answer: general-purpose engines don’t tune for a 2016 card anymore, so I wrote one that only does one thing: run one model family on P100s. Same output quality as llama.cpp, 40–50% faster plain decode, 38% faster prompt processing, and about 118 tok/s with speculative decoding. It went from first working version to live server in about a week, after two weeks of patching llama.cpp, with a lot of dropped ideas along the way.
Where llama.cpp topped out
By early October I had a llama.cpp fork with about 20,000 lines of P100 patches: other people’s sm_60 fixes, my own decode kernels for K-quants, and a faster prompt-processing path. A 27B model split across two P100s went from 28.8 to 32 tok/s. Each new kernel was worth a few percent, and the gains were getting smaller.
The reason is structural. llama.cpp runs dozens of model families on every GPU from the last ten years. Its hot paths are tuned for cards with tensor cores and fast integer dot products. The P100 has neither, but it does have fast fp16 math (2x fp32) and 732 GB/s of HBM2. To use those well you have to give up generality.
What “my own engine” means
A C++17 and CUDA program that only:
- runs one model family (Qwen 3.8, the 27B dense model and later its MoE sibling),
- on Pascal (sm_60) only,
- split across 2 (or 4) cards with tensor parallelism, weights resident in VRAM,
- and speaks the same HTTP API as
llama-server, so my tools didn’t notice the switch.
It reads the same GGUF files. Every quant type decodes bit-exactly against the reference Python decoder, and I checked output quality against llama.cpp on the same file: mean KL divergence 0.0495 versus llama.cpp’s 0.0513, a difference smaller than the noise. Same quality, not a shortcut.
The first version was correct and took 15 seconds per token. Everything after that was speed work.
What actually made it fast
In order of how much they mattered:
- One CUDA graph per decode step. The whole step on both GPUs, about 1,570 operations, is recorded once and replayed. On a card this old, launch overhead between small kernels is a real cost.
- Kernels written for how Pascal issues instructions. At batch size 1 the P100 runs out of instruction slots before it runs out of memory bandwidth. Decoding each weight block once and reusing it, with fp16 multiply-add chains (
HFMA2), mattered more than squeezing memory reads. - The model’s own draft head (MTP). Qwen 3.8 ships an extra block that guesses the next few tokens. The engine checks three guesses in one pass, which takes about as long as one normal step. Plain decode at 49 tok/s became 117–119 tok/s on benchmarks, and the output is identical to normal greedy decoding. In real chats, where guesses land less often, I see roughly 90–107 tok/s.
- A trimmed quant mix. Dropping the Q3_K tensors from the mix gave +1.5% speed and better quality (KL divergence 0.042 versus 0.071 for plain Q4_0).
- A lean attention kernel for long context. +6.6% decode at 64K and +10% at 128K, with bit-identical output.
- Prompt caching. Not a benchmark win, but the biggest one in daily use. An agent’s 100K-token first turn takes minutes; with the cache, the next turn takes seconds.
What didn’t work
I kept a rule: every change must give bit-identical output or prove the quality cost, and must beat the old build in an A/B test. A lot didn’t:
| Idea | Result |
|---|---|
| Fusing dequantization into the prompt GEMM | −5.5% |
| Prefetching in the attention kernel (3 versions) | −2% to −12% at 128K: more registers, fewer active blocks |
| Fused attention combine step | −16.8% at 32K |
| Hand-written PTX for the inner loop | 4–6% slower than the compiler’s |
| Tree drafting / EAGLE-style speculation | each branch costs +4 ms; the simple 3-guess draft already landed ~95% on benchmarks |
| Changing draft length with context depth | 3 drafts won or tied at every depth |
| q4_0 KV cache to go faster at depth | ~1% faster, not worth the quality cost |
| Pushing partial results GPU-to-GPU instead of pulling | +1.7% without drafts, 0% with them, and a race condition |
The lesson I keep relearning: measure before believing a profiler. One profiler made kernel launches look like 1.5 ms each; the real cost was about 14 µs, which killed an idea before I spent a week on it.
Should you do this?
Probably not, honestly. It made sense for me because:
- I run one model all day as a shared server, so a 50% gain pays back daily;
- the P100s were cheap (about $80 each) and aren’t going anywhere;
- I wanted to learn how inference really works on a GPU.
If you run different models every week, a patched llama.cpp with the settings from the speed series gets you most of the way for zero effort. The engine isn’t public; this post is about what’s possible on old cards, not a download.
How I ran it
- Two Tesla P100 16 GB in one machine, CUDA 12.9, power cap 160–175 W during these tests
- Decode:
tg128-style runs at short context, greedy. MTP numbers are from benchmark prompts; real chats vary. - Prompt: 2048-token prompt, q4_0 KV cache, 556–558 tok/s; 427 tok/s at 32K depth, 348 tok/s at 64K
- llama.cpp numbers: my patched fork on the same cards, best settings I found
- Quality: KL divergence against Q8_0 logits on about 11,000 tokens of chat text
- Not checked: correctness past about 128K tokens