P100 power caps, clocks and cooling: what the numbers say

2026-10-25 · Local AI · Guides · Tesla P100 16GB x2

Illustration of a large fan in a shroud beside a thermometer, with air currents drawn as lines — Illustration: AI-generated locally with FLUX.2 klein.

P100 clock under load at each power cap (MHz)

125 W cap974

150 W cap1120

Dashed line: Max boost 1328 (1328 MHz)

Short answer: the P100’s power cap goes from 125 W to 250 W. At 125 W it holds about 960–990 MHz under load; at 150 W about 1,110–1,130 MHz; at 250 W two cards in a desktop case got to 79 °C and started throttling. I settled between 150 and 175 W, with a fan curve driven by the hotter card.

The P100 is a datacenter card: no fan of its own, designed for a server chassis pushing air through it. In a desktop you have to supply the airflow, and you have to decide how much power to give it.

Power: the range is 125–250 W

nvidia-smi -i 0 -pl 150

125 W is the minimum. I tried 100 W and nvidia-smi refused it.
The cap doesn’t stick. It resets to 250 W when the driver reloads or the machine reboots, so set it at startup (a scheduled task on Windows, a systemd unit on Linux).
Something else may undo it. I had a script that lowers the cap when idle and raises it under load. A plain nvidia-smi -pl got silently overwritten by it a few seconds later. If your cap won’t stay, look for your own automation first.

What each cap does to clocks

Two P100s, under a real model load, measured with:

nvidia-smi --query-gpu=clocks.sm,power.draw,temperature.gpu,clocks_throttle_reasons.active --format=csv -l 1

Cap	Clock under load	Temp	Throttle reason
125 W	961–987 MHz	70–71 °C	power cap only
150 W	1,113–1,126 MHz at ~140 W	72–74 °C	power cap only
250 W	up to 1,328 MHz, then throttled	79 °C	heat

The throttle-reason column is the useful part. At 125 and 150 W the only reason was SW Power Cap: heat was not the limit, power was. At 250 W heat took over.

What I haven’t measured cleanly is tokens per second at each cap. On a single card with llama.cpp, 125 W versus 250 W cost about 3% decode speed. Decode on a P100 is limited more by memory than by core clock, so lower caps hurt less than the clock numbers suggest. Prompt processing is compute-bound and suffers more.

Locking clocks: -lgc doesn’t work, -ac does

nvidia-smi -lgc 1328
Setting locked GPU clocks is not supported for GPU ...

Pascal doesn’t support locked clocks. Use application clocks instead (memory clock first, it’s fixed at 715 MHz on the P100):

nvidia-smi -ac 715,1328

The default application clock was 1,189 MHz. Raising it to 1,328 lets the card boost higher when the power cap allows; under a 150 W cap it still sits around 1,120 MHz. Like the power cap, -ac resets on reboot.

Idle power: about 40 W per card

The P100 tends to sit in its full-power state (P0) even when idle, at about 40 W each. Two cards idling all day is about 700 kWh a year. Dropping the cap to 125 W when nothing is running doesn’t remove that, but it caps the worst case. Pulling the cards when you’re not using them is the only real fix.

Cooling: check which header the fan is on

Both of my cards are cooled by one 120 mm fan in a 3D-printed shroud. I had a fan curve set up, and during a long run of 100K-token prompts both cards hit 79 °C, one dropped to 645 MHz, and the fan never sped up.

The curve was on the wrong fan header. It was driving a different fan, the shroud fan had no curve at all, and the curve only read the first GPU’s temperature. Things I’d do from the start:

Find the header physically. Set one header to 100% and listen, or watch its RPM.
Drive the curve from the hotter card, not GPU 0.
Don’t trust a very low floor. At 20% one fan read 0 RPM and another gave nonsense readings. My floor is now 35%.
Make the curve steep near the throttle point. The P100 starts pulling clocks in the high 70s, so mine goes 35% up to 40 °C, 45% at 70 °C, and 100% at 75 °C.

On Linux the motherboard fan chip on many boards is a Nuvoton NCT67xx. Load the driver (sudo modprobe nct6775, and add it to /etc/modules-load.d/ to keep it), then a small script can read nvidia-smi every 2 seconds and write the fan PWM under /sys/class/hwmon/. Find the hwmon device by name: its number changed between reboots on my box.

With that curve and a 150 W cap, long runs stay around 70–74 °C with no thermal throttling.

Checklist

Set the power cap at boot, and check nothing else changes it.
Watch clocks_throttle_reasons.active under real load: power or heat?
Use -ac, not -lgc, to raise the clock target.
Make sure the fan curve is on the fan that actually cools the cards, driven by the hotter one.
Expect about 40 W per card at idle.

How I ran it

Two Tesla P100 16 GB, passive, one shared 120 mm shroud fan, desktop case, room temperature not recorded
Clock and temperature readings: nvidia-smi once a second during model inference; each row is a few minutes of one session, not a controlled sweep
Not measured: decode and prompt speed on two cards at each cap

Next: what else two P100s can do: video generation and fine-tuning.