Local AI and GPUs
Running big models on cheap, secondhand and mismatched GPUs: how to split them, which settings move tokens per second, and what still runs on old cards.
Start with a series
All 12 posts
- Why I wrote my own inference engine for two P100s
Patching llama.cpp got a 27B model to 32 tok/s on two P100s. A from-scratch CUDA engine built for just this card and model reached 49 tok/s plain and about 118 tok/s with the model's own draft head.
- P100 power caps, clocks and cooling: what the numbers say
The P100 is a passive 250 W card. What a 125, 150 and 250 W cap does to its clocks and temperatures, why nvidia-smi -lgc fails on Pascal, and the fan mistake that let both cards hit 79 °C.
- Beyond chat: video generation and fine-tuning on two P100s
Two P100s can make a short AI video clip and can fine-tune a 27B model. Both work. Neither beats a single RTX 3070 for video, and fine-tuning is measured in days.
- Two P100s in one Ubuntu box: driver 580, CUDA 12.9 and the IOMMU trap
Moving both P100s into one Ubuntu 26.04 machine. The setup was easy except for one thing: GPU-to-GPU copies ran at 0.14 GB/s until I changed one kernel setting.
- Stretching context to 500K on 8 GB cards, and why I backed it out
YaRN makes huge context windows easy to turn on. On small cards it's the KV cache, not the setting, that decides whether it works.
- Pairing a P100 with a gaming GPU: splits, buffers and the small-card trap
How to split one model across a 16 GB P100 and an 8 GB RTX 3070, and why the smaller card sets every limit.
- CUDA 13 and PyTorch dropped the P100: what still works
'No kernel image is available for execution on the device.' Here's which llama.cpp and PyTorch builds still run on sm_60.
- Is a used Tesla P100 worth it for local LLMs in 2026?
Around $89 for 16 GB of HBM2. It's the cheapest VRAM you can buy, as long as you can live with the catches.
- Q4_0 vs Q5 on a 27B: prompt speed is basically a wash
I assumed the bigger quant was costing me prompt speed. Fifty production logs said otherwise.
- Splitting a 27B model across mismatched GPUs: tensor vs layer vs RPC
An RTX 3070 plus a Tesla P100 in one box beats adding more GPUs over the network. Here's why, with numbers.
- The one flag that made prompt processing 67% faster (and why I lost it)
Raising llama.cpp's ubatch took a 27B model from 171 to 286 tok/s on long prompts. Then it quietly disappeared from my launchers.
- +65% decode on an 80B MoE from two cheap changes
Quantizing the K cache to q4_0 and updating llama.cpp took an 80B MoE coder model from 12 to 20 tok/s.