The one flag that made prompt processing 67% faster (and why I lost it)
-b 512 / -ub 256171
ubatch 320204
-b 8192 / -ub 2048286
- MODEL
- Qwen3.8-27B Q4_0
- HARDWARE
- RTX 3070 8GB + Tesla P100 16GB
- CONTEXT
- 7.8K-token prompt
- RESULT
- 286 tok/s prompt processing
- WHAT CHANGED
- batch 8192 / ubatch 2048 (was 512 / 256: 171 tok/s)
If your agent workloads send big prompts (tool definitions, files, long chat history), prompt processing matters more than decode speed. On a 3070 + P100 split, one setting mattered more than anything else I tried.
The A/B
Same model, same tensor split (7.0 / 15.5), same 7,826-token prompt:
-b / -ub |
Prompt tok/s |
|---|---|
| 512 / 256 | 171 |
| ubatch 320 (later test) | 204 |
| 8192 / 2048 | 286 |
That’s +67% just from letting llama.cpp process the prompt in bigger chunks.
Why it disappeared
A month later every launcher was running ubatch 128. Nobody decided that on purpose. It crept in while squeezing in 150K context plus a vision projector. A bigger ubatch means bigger compute buffers: roughly +140 MiB per +128 of ubatch, on every GPU. On a mismatched pair the smallest card sets the limit, and the 8 GB 3070 had about 530 MiB free.
Takeaways
- Benchmark
-ubbefore you touch anything else for prompt speed. - On mismatched GPUs, shift the tensor split toward the bigger card to free buffer room on the small one.
- Comment the reason for every launcher flag. Settings drift when nobody remembers why they’re there.