+65% decode on an 80B MoE from two cheap changes

2026-05-13 · Local AI · Experiments

Illustration of a grid of small modules, a few lit in orange, wired to a memory rack — Illustration: AI-generated locally with FLUX.2 klein.

Decode speed, Qwen3-Coder 80B at 131K context (tok/s)

Short prompt, before12

Short prompt, after20.1

Long prompt, before13.6

Long prompt, after22.5

MODEL: Qwen3-Coder 80B
HARDWARE: Consumer GPU + DDR4 offload
CONTEXT: 131K
RESULT: 20.1 tok/s short prompt, 22.5 tok/s long prompt
WHAT CHANGED: K cache q8_0 to q4_0, --cache-reuse 256, llama.cpp b9006 to b9133

Big MoE models on consumer hardware usually spill into system RAM, and then memory bandwidth becomes the bottleneck. Two config changes and a llama.cpp update gave the biggest speedup I’ve had without new hardware.

Results

	Before	After	Change
Short prompt decode	12.0 tok/s	20.1 tok/s	+67%
Long prompt decode	13.6 tok/s	22.5 tok/s	+65%

What changed

--cache-type-k q4_0 (was q8_0). At 131K context the KV cache is huge. Halving the K cache frees memory and, more importantly, DDR4 bandwidth that weight reads were competing for.
--cache-reuse 256, so repeated prompt prefixes such as agent system prompts and tool lists aren’t reprocessed.
A newer llama.cpp build, which brought MoE routing optimizations and better CUDA kernels.

Caveats

These are combined numbers, so I can’t say how much each of the three changes contributed. Back up your working binaries before updating: newer isn’t guaranteed to be faster on old cards.