The invisible weight of a prompt: what AI really consumes

The other day, out of plain curiosity, I asked an AI a silly question: how much power does a GPU draw? Not for a project, not for a class: just because I realized I had no idea. I spend my days sending prompts, and I had never given a second's thought to what was burning behind them.

The answer cooled me down a little. A single recent datacenter card (an H100) can draw up to 700 W at full load. A node of eight cards is close to 8 kW: enough to power a small house, just to run a single model.

And then the engineer's reflex kicked in. In I&C, in V&V, we spend our time sizing, measuring, asking what a system consumes and with what margin. Yet for the tool I use the most, I had never applied that reflex. As if the prompt were free. As if it weighed nothing.

Spoiler: it does weigh something. And that is what I want to talk about.

I. The cloud illusion: a prompt is never free#

The word “cloud” is an absolute marketing triumph. It evokes something airy, immaterial, infinite. But behind the cloud there is metal, silicon, and above all electricity. A lot of electricity.

First, two regimes need to be told apart, because they have nothing in common:

• Training: building the model. Thousands of GPUs run for days, even weeks. It is colossal, but it is a one-off.
• Inference: using the already-trained model. This is what happens every time I send a prompt. Taken in isolation, it is much lighter: around 0.24 Wh for a text request on optimized infrastructure.

0.24 Wh is ridiculous. A few seconds of an LED bulb. Except that this figure is a trap, because you never send just one prompt. We send billions of them every day, across the planet. And that is where the ridiculous becomes massive: what matters is not the unit cost, it is the cost multiplied by an entire world that prompts nonstop.

An engineer knows the raw-figure trap well, by the way. A card rated at 700 W TDP does not draw 700 W all the time: it depends on the load, the clock frequency, the batch size. On one measured 8×H100 node, peak readings were around 8.4 kW while the theoretical capacity was close to 10 kW. Max power ≠ actual power. And GPU power alone ≠ system consumption: you have to add cooling, power supply and the rest of the infrastructure. The real bill is always steeper than the “GPU” line on the spec sheet.

II. On the hardware side, efficiency is not a detail#

The good news is that this cost is not inevitable. And the part that interests me most, as a future engineer, is what can be done at the hardware level to reduce it. There are several levers, and some of them are spectacular.

Lower the precision. This is the most powerful lever. A computation in FP32 (32-bit floating point) is expensive; the same computation in FP16, INT8 or INT4 uses much simpler arithmetic units, and therefore much less energy. Tensor Cores are optimized precisely for this. For inference, quantization can cut consumption by a factor of 4 to 10 without touching a single screw of the hardware. Nearly free optimization.

Cap the power (undervolting, power capping). On server GPUs, you can enforce a power limit with a simple command:

# Cap an H100 at 400 W instead of 700 W
nvidia-smi -i 0 -pl 400

You lower the voltage while keeping a stable frequency. Typical result: −10 to −30% consumption for only −5 to −15% performance. The trade-off is so favorable that optimized datacenters do it systematically. Capping is often smarter than pushing.

Manage the thermals. A GPU that runs hot throttles and loses efficiency. Liquid cooling, immersion, better heat pipes: beyond comfort, this is efficiency. The gains are indirect (fewer fans, more stable operation), around 5 to 15%, but they add up with the rest.

Change the hardware. A general-purpose GPU is not the most efficient option for inference. Specialized chips (ASICs such as TPUs, NPUs) can consume 5 to 20 times less than a conventional GPU on a specific workload. This is the most radical lever, but also the least flexible: an ASIC does one thing, very well, and nothing else.

Don't waste the rest. Two items people forget: memory and scheduling. VRAM (HBM) can account for 20 to 35% of the GPU's energy, hence the value of tuning batch sizes, exploiting sparsity and compressing weights. And leaving eight GPUs on for a task that needs one is throwing energy out the window: dynamic scheduling and putting idle components to sleep recover that waste.

If I had to sum it up in one sentence: the biggest gains come from specialized hardware, combined with quantization and power capping; thermals and memory come next, as a complement.

III. The real issue: restraint in the face of the race for power#

Except there is a catch, and it is the most uncomfortable one. All these efficiency gains risk being swallowed up by scale.

It is an old paradox, the Jevons paradox: when a resource becomes more efficient to use, we don't consume less of it, we consume more. Dividing the energy of a prompt by ten will not reduce the global bill if, at the same time, the number of prompts is multiplied by a hundred. And that is exactly the current trajectory: the more efficient it gets, the more accessible it becomes, and the more we put it everywhere.

So hardware efficiency is necessary, but not sufficient. The real question comes upstream: do we need this computation at all? Do we really need to run a giant model on a task that a small model, or even a script, would handle perfectly well? We are back to engineering logic: size things right, don't oversize for comfort.

And this is where it concerns me directly. In a few years, it is my generation that will design, deploy and size these systems. If we treat energy as an invisible variable (something that gets sorted out “elsewhere”, at the hosting provider), we will repeat the same mistake as everywhere else: optimize performance, ignore the consequence. Energy efficiency should be a design constraint, just like latency or cost. Not a box you tick after the fact.

Conclusion: thinking about AI like an engineer#

I am not going to stop using AI: that would be absurd, and I would be the biggest hypocrite of all. But ever since that silly question about how much a GPU consumes, I no longer see the prompt the same way. It is not an immaterial request heading off into a cloud. It is current flowing through silicon, somewhere, right now.

Thinking about AI like an engineer means no longer treating it as magic. It means asking three simple questions: what do I really need, what is the real cost, and can I do the same for less? Measure before scaling. Prefer a right-sized model to a giant one. Know that a tiny number, multiplied by the whole world, becomes enormous.

A while ago, I wondered what it cost my brain to delegate too much to AI. Today, I am asking the same question for the planet. Same reflex, different scale: behind every convenience there is a cost. An engineer's job is precisely never to lose sight of it.

What we don't measure, we always end up paying for.