The second scaling law: the frontier moved to inference

For a decade you bought intelligence by the parameter. Now you buy it by the token.

From roughly 2018 to 2024, progress in AI ran on one reliable lever: make the pretraining run bigger. More parameters, more data, more GPUs, a better model. It worked so consistently that it became the whole strategy, and the field learned to measure ambition in FLOPs and parameter counts.

That lever has not broken, but it has started to bend. Researchers now argue that the returns to ever-larger pretraining are diminishing, among them a 2025 MIT group whose paper carries the blunt title “Meek Models Shall Inherit the Earth.” At the same time a second lever appeared, one that adds capability after training is finished. Let the model spend more compute at the moment it answers.

Thinking is the new scaling

In 2024, work from Google DeepMind and UC Berkeley showed that a model allowed to think longer at inference, sampling several attempts and checking its own working, could match or beat a model up to an order of magnitude larger, in some settings. Around the same time, researchers at Stanford found that simply letting a model make more attempts at a hard problem kept lifting the share it eventually solved, across four orders of magnitude of extra compute. The behaviour had a name: test-time compute.

The clearest proof arrived in Nature in September 2025. DeepSeek’s R1, trained by reinforcement learning to reason, learned to write out its working, often thousands of tokens of it, before committing to an answer. On a hard mathematics benchmark its score rose from 15.6% to 71%. The frontier had found a new direction, and it pointed at inference.

The bill lands on inference

None of this is free. A reasoning answer is a long answer, and a long answer costs energy. Microsoft researchers put a number on it, in peer-reviewed work published in Joule in April 2026: a typical frontier query used about 0.31 watt-hours, while a long reasoning query, running roughly fifteen times longer, used 3.91. That is thirteen times more energy for a single question. They also traced the aggregate effect. If just one query in ten becomes a long-reasoning one, total data-centre demand for that workload more than doubles.

It compounds, because inference is already where the compute lives. Inference now accounts for the majority of AI compute in deployment, and its share is still climbing; Deloitte put it at roughly two-thirds for 2026. Training is a one-time cost for each model. Inference is paid on every token, on every query, for the life of the model. The reasoning shift takes the part of the workload that already dominated and makes each unit of it heavier.

Straight into the memory wall

Here is why this reaches all the way down to the silicon, and why it is structural. Generating a token from a large model is mostly an act of reading. To produce each one, the machine streams the model’s full set of weights out of memory and consults a growing cache of the conversation so far. The arithmetic on top is comparatively cheap. The reading is not.

Reaching off-chip to memory costs on the order of a hundred to a thousand times the energy of the calculation it feeds, a ratio the field has understood since Mark Horowitz’s 2014 analysis and still cites today. So token generation is governed by memory bandwidth, and the compute units spend much of their time waiting for it.

Reasoning multiplies precisely this operation. Every extra thinking-token is another full read of the weights and another pass over a longer cache. The new scaling axis, the one the whole field is now climbing, runs directly along the most expensive thing a conventional chip does. Adding raw compute does little, because the bottleneck was never the compute. A widely cited 2024 analysis, “AI and Memory Wall,” makes the trend plain: server compute has scaled far faster than the memory bandwidth that feeds it, and the gap is still widening.

The substrate that moves least

If the cost of thinking is the cost of moving weights, then the chip that suits the reasoning era is the one that barely moves them. That is the whole idea behind computing inside the memory. Phynomy stores each weight as a physical resistance in a crossbar and performs a model’s core operation in place, so the weights stay where they sit and the answer forms around them. The number that matters then is tokens per joule: how much thinking you get for each unit of energy spent.

The honest state of that work is worth stating plainly. The device at the centre of it, a dual-doped memristor, is patented, fabricated and measured. The full chip is active research and has not shipped. The headline figure, on the order of thirty times more useful work per unit of energy, is a design target drawn from simulation and independent research; it is not yet a measured benchmark. What the outside evidence supports is the direction, and the direction has become hard to miss.

For years the defining question for AI hardware was how large a model you could train. The question now is how cheaply a model can think. That one is answered in the memory.