Splitting a 27B model across mismatched GPUs: tensor vs layer vs RPC

· Local AI · Experiments · RTX 3070 8GB, Tesla P100 16GB, GTX 1660 6GB

Illustration of a glowing cube split across two boxes, with a third box far away on a long cable
Illustration: AI-generated locally with FLUX.2 klein.
Decode speed, Qwen3.8-27B Q5 (tok/s)
3070 + P100, tensor split9.9
3070 + P100, layer split7.8
+ remote P100 (RPC)5.91
+ remote P100 + 16604.32

Every remote GPU added over 1GbE RPC made decode slower.

MODEL
Qwen3.8-27B Q5
HARDWARE
RTX 3070 8GB + Tesla P100 16GB
CONTEXT
111K
RESULT
9.9 tok/s decode, 83 tok/s prompt (tensor split)
WHAT CHANGED
-sm tensor instead of -sm layer (7.8 decode / 89 prompt)
MODEL
Qwen3.8-27B Q5
HARDWARE
RTX 3070 8GB + Tesla P100 16GB + remote P100 over 1GbE
RESULT
5.91 tok/s decode (layer split over RPC)
WHAT CHANGED
Added a second machine's P100 over the network

The question: I have an RTX 3070 (8 GB) and a Tesla P100 (16 GB) in one desktop, plus another P100 and a GTX 1660 in a second machine on the LAN. What’s the fastest way to run a dense 27B model at Q5 with long context?

Short answer: keep it in one box and use tensor split. Everything involving the network was slower.

Local: tensor split wins decode

llama.cpp, flash attention on, weights split 6.5 / 16 between the 3070 and the P100, real llama-server at 111K context:

Split mode Decode (tok/s) Prompt (tok/s)
-sm layer 7.8 89
-sm tensor 9.9 83

Tensor split is +27% on decode for a small hit to prompt processing. It still fits at 160K context (9.3–9.9 tok/s); 200K doesn’t fit.

-sm row doesn’t work on current builds at all (“does not support split buffers”).

Adding GPUs over RPC: layer split

llama.cpp’s RPC backend lets you add remote GPUs. With layer split, every extra remote card made things slower:

GPUs Decode (tok/s)
Local 3070 + P100 7.79
+ remote P100 5.91
+ remote P100 + 1660 4.32

Best RPC arrangement was putting the remote device first in -dev so the logits stay local: 7.33 tok/s at under 0.5 MB/token. Faster links barely help: simulated 2.5G gets ~7.47, 10G ~7.55. The problem is latency per hop, not bandwidth.

Tensor split over RPC: technically works, practically useless

Getting it to run at all took two fixes:

  1. The RPC client overflowed its stack; rebuilding with a 256 MB stack (editbin /STACK) fixed it.
  2. A zero-byte recurrent-state tensor view tripped a bounds assert in ggml-rpc.cpp’s deserialize_tensor. Skipping the check when nbytes == 0 fixed it.

Then the numbers:

Setup Decode (tok/s)
P100 + remote P100 0.60
3 GPUs 0.88
4 GPUs 0.19

Tensor mode moves about 150 MB per token over the wire: this model’s recurrent state (~3 MB × ~48 layers) can’t be split, so it gets shipped around. On 1GbE that’s hopeless. Back-of-envelope: you need >13 Gbps just to stop being bandwidth-bound, and something like 40 Gbps with RDMA to match local layer split.

Hybrid: local tensor + remote layers

The ideal would be tensor split locally with a few layers offloaded to the remote box via -ot. It fails with “pre-allocated tensor in a buffer that cannot run the operation”; in tensor mode llama.cpp only schedules its meta device. Even if patched it wouldn’t be faster, only more context (~200–260K).

Takeaways