Intelligence
per Joule

There is no agreed definition of “intelligence per watt.” The term is used loosely because it is not pinned down: NVIDIA reports output tokens per joule, Epoch AI reports capability per FLOP, and most comparisons lean on benchmark scores. A token is not intelligence, and raw throughput per watt systematically overstates real progress — a model streaming 500 tokens a second can be worse per task than one streaming 40.

The quantity that matters is the product of two measurable figures, and neither is published as an “intelligence per watt” number. The first is hardware efficiency: how many useful operations one watt of silicon performs. The second is algorithmic efficiency: how much capability each operation buys, which is why a 2024 model beats a 2022 model on the same chip. Both are measurable, and their product is what has been rising.

The measure nobody agrees on

The reason there is no single figure is that the two ingredients are measured on different axes and neither is quality-adjusted. NVIDIA measures throughput: output tokens per joule. Epoch AI measures capability per FLOP. Vendors publish benchmark scores that are chosen after the fact. None of these is the quantity “intelligence per watt” names, and none is directly comparable to another.

The practical consequence is that raw throughput per watt systematically overstates real progress. A model that streams five hundred tokens a second can be stupider than one that streams forty — cheaper per token, worse per task. This is the single most important caveat in the piece, so it comes first.

A defensible version of the metric is quality-adjusted: how many useful answers per joule, with “useful” anchored to a fixed difficulty rather than a token counter. Nobody publishes that. But the two ingredients are measurable, and that is enough to make the point.

The silicon got cheaper per watt

Start with the solid, measurable part. If you line up NVIDIA’s flagship data-centre parts and divide peak dense compute by power draw, the trend is unmistakable. Dense, non-sparse figures are used throughout, because the sparse “structured” numbers are marketing multipliers that do not apply to real weights.

On the same dense, non-sparse axis — comparing like for like — that is roughlyan order-of-magnitude increase from A100 to B300 in six years, and part of the gain arrived only with FP4, which NVIDIA did not ship until Blackwell. Compute got cheaper per watt and cheaper per bit of precision.

Watts alone are only half the story, and NVIDIA knows it. The company now pitches efficiency at the system level, in tokens per megawatt — which is closer to the truth than per-GPU TFLOPS, because a GPU does nothing useful on its own. The GB300 NVL72 rack-side claims against the Hopper generation are where the “tokens per watt” framing lives:

These are vendor numbers, not measurements. 5× TPS per megawatt and50× AI-factory output are NVIDIA’s figures against their own Hopper generation, on their own benchmarks, in their own white paper. They are directionally right and precisely optimistic. Take them as the shape, not the figure.

The algorithms got cheaper too

The hardware curve is the less interesting of the two. The larger multiplier is thatthe same silicon now does more thinking per FLOP, because the models changed.

The clearest change is Mixture-of-Experts — a model that stores a large number of parameters but only wakes a small slice for each token. DeepSeek-V3 is the example: 671 billion parameters in total, about 37 billion active per token. This matters. It means the modelcosts like a roughly 37B dense model at inference while behavinglike a frontier model — and a lot of what looks like “bigger” is not what you pay for.

The pattern is everywhere now. Qwen3-30B-A3B stores 30B but runs 3B. Qwen3-235B-A22B stores 235B, runs 22B. Mixtral 8x7Bstores roughly 47B, runs roughly 13B. GLM-4.5 stores roughly 355B, runs roughly 32B. A dense Llama 3 405B has no escape hatch: it runs all 405B, every token.

MoE buys cheap math, not cheap memory. The full 671B still has to sit in RAM or VRAM — that is the hard wall on the machine that can hold it. But the thing you pay for every token is the active slice, and that is the number that drives the bill.

Then there is quantization. Skipping from FP16 to FP8 halves the bytes; skipping to FP4, which Blackwell made mainstream, halves them again. NVIDIA markets FP4 at roughly2× the throughput and about 2× the energy efficiency versus FP8 for the same model. The aggressive end of the road, ternary roughly 1.58-bit weights — the BitNet line — pushes the idea that most of a large model can ride on just −1, 0 and +1. That is the same trick that let a roughly 770B ternary model run on a laptop earlier this year. Cheap precision is a real lever and it compounds with sparse activation.

What a joule buys, right now

Put the two curves together and the result is that cost per unit of capability has collapsed. DeepSeek-V3 is the cleanest data point. From its own technical report: 671B total / 37B active, trained on 14.8 trillion tokens, 2.788M H800 GPU-hours, at a stated roughly $5.576M. That figure is the report’s assumed cost at $2 per GPU-hour — a modelled number, not an invoice. GPT-3, in 2020, was a 175B dense model at roughly 1,287 MWh and roughly $4.6M, where the energy and dollar figures are again estimates from the published literature. DeepSeek is more capable per FLOP and came in at a comparable dollar figure because MoE keeps the active FLOPs low.

So “intelligence per joule” is really two numbers multiplying: silicon per watt and capability per FLOP. On their own they are measurable. The product is what keeps doubling, and it is why the same money now buys a reasoning model that would have been science fiction a few years ago.

The cost of cognitive work

What does this mean in practice? The chart below sets out the cost of a unit of cognitive work — one hour of output at roughly a competent, “130 IQ” level — and how that cost changes over time. The left half is anchored to measured data: the wage of a cognitive worker (BLS), and today’s raw inference pricing. The right half is an extrapolation.

The way to read it: today, a competent cognitive worker costs roughly $20–30 per hour (BLS median is about $23 per hour; a professional role closer to $30; fully-loaded, including benefits, about $45). An AI can already produce much of the work, but producing verified output — output you can trust enough to ship — still costs about the same as the wage, because a human is in the loop checking the result. That is the flat part of the curve through today.

The extrapolation is where the argument lands. As verification and reliability improve, the human checks fall away and the cost converges on the raw compute cost — which is already near $0.10 per hour of reasoning for a mid-tier model, and falling. If that holds, the endpoint is roughly $0.01 per hour: a unit of cognitive work that today costs as much as a worker, and within a decade costs about a cent. The whole piece can be read as a scenario with a direction rather than a forecast with a number.

The second panel puts the same result in the frame of the rest of this piece: the same intelligence, per joule, rising roughly 43× by 2031 and on the order of 200× by 2036 if the efficiency curves hold. The cost line and the efficiency line are the same result measured two ways.

2031 — five years out

Everything from here is a scenario, not a measurement. These are qualitative bets, and the caveats follow.

Five-year bets:

• A frontier-class reasoning model on a workstation. Per-watt efficiency keeps climbing at roughly the historical rate for a while, so a single roughly 500 W workstation GPU runs a model that beats today’s frontier on day-to-day work. “Frontier” and “local” stop being opposites.
• Reasoning becomes the default, and it is charged per thought.Test-time compute — the trick where a model spends extra tokens to think before answering — stops being a novelty. A model that thinks for a second uses more energy than one that answers instantly. The energy story stops being about throughput and becomes about deliberation.
• Another roughly 10–30× drop in cost per unit of capability.FP4 and ternary become table stakes; dense 400B-scale training becomes a niche. The people still scaling dense models are the ones without a data constraint — and they are paying for it.
• Energy, not chips, is the binding constraint. Power draw per rack has gone from roughly 40–60 kW air-cooled to roughly 100–200+ kW liquid-cooled, and vendors are chasing 300 kW+. Data-centre electricity has been climbing as a share of grid demand, and everyone is now fighting over the same megawatts. The scarce input is the grid.

2036 — ten years out

Ten years out the curve bends, and here is how.

• Per-watt up roughly 20–50× from today, but slower than the recent past. The silicon is hitting walls — packaging, power delivery, and the fact that compute is getting cheap but memory bandwidth is not. Each token re-reads weights; the bottleneck is bandwidth, not FLOPs. Raw per-watt FLOP scaling decelerates, and the wins shift to sparsity, precision, and architecture.
• “Intelligence per joule” becomes a strategic metric. Not a benchmark researchers publish — an economic and geopolitical one. When a country or a company can produce a useful answer for a fraction of the joules, that is the same as producing it for a fraction of the money and a fraction of the carbon. Efficiency becomes the competitive axis, alongside capability.
• Always-on personal intelligence. A home box — or a laptop on a plugged-in desk — runs reasoning that today needs a data centre. “Assistant” stops being pinned to a subscription; the model is yours, and so is the electricity bill. That changes who owns the intelligence.
• The metric question gets settled. Raw tokens per watt will not survive scrutiny, and quality-adjusted capability per joule — how manycorrect, useful answers per unit of energy — becomes the number people quote. A 2-bit model that is 2× cheaper per token but needs 3× more tokens to finish a task is a net loss, and by 2036 everyone will have learned that.

What it means

If the shape holds, the interesting consequence is not “AI gets smarter” (it will). It is that energy becomes the unit of intelligence. Right now intelligence is priced in dollars and measured in parameters. If the efficiency curves hold, it gets priced in joules: the cost of a thought approaches the cost of the electricity that makes it.

That has an upside and a risk. The upside: thinking gets cheap the way computation got cheap, and the barrier to owning intelligence — not renting it — falls to the price of a decent GPU and a power outlet. That is the prize for those who run models at home. The decade ahead is the one where that stops being a hobby and becomes normal.

The risk: cheap, fast, low-quality tokens are dangerous. The moment output is nearly free, the incentive is to generate a lot of it and trust the tool to sort it out. Per-watt metrics that ignore quality will reward that. The measure that matters is the useful answer per joule, not the token per watt.

Which line wins — the one for cheap thinking or the one for cheap, unchecked output — depends on which number gets watched. The “per watt” denominator is the side that keeps surprising people. The “intelligence” numerator is the side that is easy to fake with a faster token stream. The metric to track is whether the answer was worth the electricity.

Credits & sources

Figures were checked against primary sources where possible. Vendor claims and schematic projections are labelled as such.

• NVIDIA datasheets — A100, H100, H200, B200, B300/Blackwell Ultra (peak dense TFLOPS, TDP, memory, bandwidth). The A100 “156 TFLOPS” figure is the TF32 value; the FP16 dense value is 312. B300 TDP is an estimate.
• NVIDIA GB300 NVL72 — “10× user responsiveness (TPS per user), 5× throughput (TPS per megawatt), 50× AI-factory output vs Hopper” — vendor claim, presented as an upper bound.
• DeepSeek-V3 technical report (arXiv 2412.19437) — 671B total / 37B active, 14.8T tokens, 2.788M H800 GPU-hours, roughly $5.576M at the report’s assumed $2/hour; MoE + multi-head latent attention. Cost is a modelled figure.
• Stanford AI Index 2025 (arXiv 2504.07139) — training compute doubling roughly every five months; inference cost of a fixed capability down roughly 280× in about two years (late 2022 → late 2024); rising data-centre power.
• Epoch AI — frontier LLM training compute doubling roughly every 5.2 months since 2020; capability-per-FLOP framing.
• BLS — median wage and employer cost (OEWS, ECEC): median around $23/hour, professional roles around $30/hour, fully-loaded (with benefits) around $45/hour.
• Scaling law — Chinchilla (arXiv 2203.15556): compute-optimal roughly 20 tokens per parameter. GPT-3 (arXiv 2005.14165) and Patterson et al. 2021 (arXiv 2104.10350) for the 3.14×10²³ FLOPs and roughly 1,287 MWh anchors.
• Model cards — Qwen3 (30B-A3B, 235B-A22B), Mixtral 8x7B, GLM-4.5, Llama 3, BitNet / ternary line.
• Author — the framing and the forecast, which is explicitly labelled as a forecast rather than a measurement.

Charts and animations generated locally with matplotlib and ffmpeg from the verified figures above. The hardware and algorithmic figures are real; the right half of the forecast is a bet, and the piece says so.