Stretching context to 500K on 8 GB cards, and why I backed it out

2026-09-27 · Local AI · Experiments · RTX 3070 8GB, GTX 1660 6GB, RTX 3060 Laptop 6GB

Illustration of a very long paper scroll running up to a small box on a table — Illustration: AI-generated locally with FLUX.2 klein.

KV cache size for a 27B at q4_0 (~17.6 KB per token) (GB)

160K tokens2.8

500K tokens8.8

1M tokens17.6

Dashed line: all of an RTX 3070's VRAM (8 GB)

MODEL: Qwen (local models)
HARDWARE: RTX 3070 8GB
CONTEXT: 458K
RESULT: Runs and answers correctly with q8_0 KV cache, 7.8 of 8 GB VRAM used
WHAT CHANGED: YaRN ×1.75 (up from 262K native)

Most models I run ship with a 262K context window. llama.cpp’s YaRN rope scaling lets you stretch that with one flag, so I did.

Round 1: 458K and 524K

Machine	Before	After	How
RTX 3070 box	262K, f16 KV	458,752	YaRN ×1.75, q8_0 K+V cache
Second box	262K, q4_0 KV	524,288	YaRN ×2.0, q8_0 K+V cache, KV in system RAM

Both verified with real completions, not just a server that starts. The 3070 sat at 7.8 of 8 GB.

Round 2: 1M everywhere

If 500K works, why not 1M (YaRN ×4)? I pushed every server to it.

It fell apart. The small cards (8 GB 3070, 6 GB 1660, and the 6 GB laptop version of the 3060, not the 12 GB desktop card) started thrashing: constantly shuffling memory instead of computing. Everything got rolled back the same day.

The math that explains it

From Part 1: the KV cache for a 27B costs about 17.6 KB per token, even at q4_0.

160K tokens → ~2.8 GB
500K tokens → ~8.8 GB
1M tokens → ~17.6 GB

That’s more than all the VRAM on any of these cards, before loading a single weight. The context setting isn’t what matters. Where the KV cache lives is.

What I run now

KV cache in system RAM (--no-kv-offload) on the small cards. The GPU holds weights and does the compute, and the cache lives in plentiful, cheap DDR4. It’s slower per token, but it doesn’t thrash.
Moderate context profiles (262K–524K) instead of the maximum.
YaRN has a quality cost: in my testing, short prompts got roughly 15–20% worse with heavy scaling. If most of your prompts are short, a huge window hurts you on every request.

Takeaways

Before raising context, multiply KV bytes per token by the context size. If it’s bigger than your free VRAM, move the cache to RAM or don’t do it.
Stretching context with YaRN isn’t free even when it fits.
Write down why you rolled something back. Months later “we never did that” and “we tried that and it broke” look identical.