Beyond chat: video generation and fine-tuning on two P100s
- MODEL
- Wan 2.2 TI2V-5B (Q8, 20 steps)
- HARDWARE
- Tesla P100 16GB + Tesla P100 16GB
- CONTEXT
- 704x1280, 73 frames
- RESULT
- 12.8 min per clip (RTX 3070 in ComfyUI: 8.5 min)
- WHAT CHANGED
- own CUDA engine, model split across both cards
- MODEL
- Wan 2.2 TI2V-5B Turbo (Q6_K, 4 steps)
- HARDWARE
- Tesla P100 16GB + Tesla P100 16GB
- CONTEXT
- 704x1280, 73 frames
- RESULT
- 4.6 min per clip (RTX 3070: 4.0 min)
- MODEL
- Qwen3.8-27B QLoRA r16
- HARDWARE
- Tesla P100 16GB + Tesla P100 16GB
- CONTEXT
- 2048-token samples
- RESULT
- ~147 s per step of 8 samples (~5 days per epoch of 22,800 samples)
Short answer: for video, two P100s ended up about 1.5x slower than one RTX 3070 (12.8 versus 8.5 minutes per clip), and close to even with a 4-step “turbo” model (4.6 versus 4.0 minutes). For fine-tuning, a 27B QLoRA runs but needs about 5 days per pass over a modest dataset. The P100’s lack of tensor cores, which barely matters for chat, matters a lot here.
After the chat engine I wanted to know what else 32 GB of cheap HBM2 is good for. I warned myself going in that these are compute-heavy jobs, which is exactly where a 2016 card is weakest. That turned out right.
Video: Wan 2.2 5B, image to video
The model is Wan 2.2 TI2V-5B, an open image-to-video model. On my RTX 3070 it runs in ComfyUI. I wrote a CUDA engine for it on the P100s, the same way I did for chat, and checked every part against ComfyUI’s own code in fp32: the text encoder, the sampler, each transformer block and the video decoder all match.
The job: one 704x1280 clip, 73 frames (about 3 seconds at 24 fps), 20 sampling steps with guidance.
Where the time went
| Stage | First version | After fixes |
|---|---|---|
| Sampling (40 model passes, split over both cards) | 8.9 min | 8.9 min |
| Video decode (84 tiles) | 8.9 min on one card | 3.2 min on both |
| Load, text, encode | under 1 min | under 1 min |
| Total | 18.6 min | 12.8 min |
Sampling came out where I predicted: about the same as the 3070. Two P100s together have roughly the raw fp16 compute of one 3070’s tensor cores, and video diffusion is almost all matrix multiplication.
The decode was the surprise. The decoder splits the video into 84 tiles. Version one did them all on one card in fp32. Two fixes cut it from 535 to 191 seconds:
- Split the tiles over both cards. Output identical, time halved.
- Use fp16 math with a careful cuBLAS setting. Pascal runs fp16 matrix math with fp16 accumulation about 2.5x faster than with fp32 accumulation (12–13 versus 4.6–4.9 TFLOPS in my tests). The default fp16 algorithm was too inaccurate (error up to 1.7%), but one specific cuBLAS algorithm (
CUBLAS_GEMM_ALGO6, which accumulates in short blocks) stayed at about 0.3% error. The decoded frames were within 61 dB PSNR of the fp32 version: no visible difference.
Turbo: fewer steps beats faster kernels
A distilled “turbo” version of the model needs 4 steps instead of 20, and no guidance pass: about 10 times less compute. On the P100 pair a clip took 4.6 minutes versus 4.0 on the 3070. At that point the decode is most of the time.
Verdict: it works and it frees the 3070, which is the real value: the gaming PC stays usable while the server makes video. As a speed upgrade, it isn’t one.
Fine-tuning a 27B model with QLoRA
QLoRA trains a small add-on (a LoRA adapter) on top of a 4-bit copy of the model, so a 27B fits in 2x16 GB. I used Unsloth with PyTorch on Linux.
What I ran into:
- PyTorch needs a build that still includes sm_60. On Linux, the cu126 wheels of PyTorch 2.14 still do. (On Windows I had to use cu118, see Part 2.) The PyTorch team has said 2.14 is the last release with prebuilt Pascal support.
- No bf16, so it trains in fp32. Unsloth forced fp32 compute for this model on the P100. That’s slower and uses more memory.
- Splitting the model by hand. The automatic “balanced” split ran out of memory on the second card, because the final output layer’s fp32 values are about 1.2 GB per example. I put the first 36 layers on card 0 and the rest, plus the output layer, on card 1.
- GGUF can’t be the starting point. The training library couldn’t load this model’s GGUF files, so I trained from the original 52 GB bf16 checkpoint and converted back to GGUF afterwards.
Speed
| Setting | Time per step (8 samples) |
|---|---|
| 4,096-token samples, automatic split | 268 s, then out of memory |
| 2,048-token samples, manual split | 147 s |
That’s about 18 seconds per sample. A full pass over my dataset (about 22,800 samples) would take about 5 days at a 125 W cap. A few tweaks (fp16 in some places) saved about 5%, nothing more.
Result
I let it run overnight: 110 steps, 880 samples, about 4% of the data, 5.3 hours. The merged model loaded and answered correctly, ran at the same speed, and was worse to talk to. I switched back to the original the same morning. With the loss flat and so little of the data seen, that’s not surprising. A useful run needs days on these cards, or a few hours on a rented modern GPU.
Verdict: possible, not practical. For fine-tuning, the P100’s VRAM gets you in the door and its compute keeps you waiting.
What two P100s are good for, after all this
| Job | Good fit? |
|---|---|
| Chatting with a 27B model | Yes. 50–120 tok/s with the right engine. |
| Long context (100K+) | Yes. 32 GB of fast memory is the point. |
| Image input on a chat model | Yes. A 2,000-token screenshot in about 6 seconds. |
| Video generation | Works, ~1.5x slower than a 3070. Useful to offload, not to speed up. |
| Fine-tuning a big model | Works, but days per pass. Rent a GPU instead. |
How I ran it
- Two Tesla P100 16 GB; video at a 125–150 W cap, fine-tuning at 125 W
- Video: same image, prompt, seed and settings on both setups; P100 times are wall-clock for the whole clip, one run each. The 3070 ran ComfyUI with its default model offloading.
- cuBLAS numbers: my own benchmark of the model’s matrix shapes, error measured against fp32 accumulation
- Fine-tuning: Unsloth QLoRA, rank 16, all attention and MLP projections, batch 1 x 8 accumulation, PyTorch 2.14 cu126, Ubuntu 26.04
- Not done: a frame-by-frame comparison of P100 and 3070 video output at full size; a full fine-tuning epoch