Useful work

Why I try to understand AI inference efficiency in order to judge how it will transform our world

There is this great chart from Our World in Data, showing solar photovoltaic (PV) module prices between 1975 and 2024. In just under 50 years, the price per watt dropped from $132 to $0.27, a staggering 500-fold reduction.¹ Technological progress and industrialisation made panels far cheaper to produce, and at the same time more efficient to operate: the best research cells have roughly doubled their conversion efficiency over the same period.² As a result, solar is now the cheapest source of electricity and its exponential growth doesn’t appear to lose any steam.

AI seems to be on an even faster track of breakneck technological progress, especially since the release of ChatGPT in November 2022. This month, we have even reached a point where the frontier labs themselves are calling for a slower pace.

Whereas this race between frontier labs is fascinating (and sometimes scary) to watch, there is another development going on which has caught my attention, as it appears to have parallels with what we have seen in photovoltaics and many other industries: a relentless race towards improved efficiency.

In engineering, system efficiency is defined as the amount of useful work done by the system per unit of energy required. For PV, this is the electrical power generated divided by the solar power influx. For wind energy, this is the electrical power generated divided by the total kinetic power flowing through the rotor plane. It is a key driver of cost-effectiveness of new technologies.

Now, in the world of AI, system efficiency is not as clearly defined as one might hope. Most public benchmarks present energy required per token, or set of tokens. While these benchmarks are useful for comparing different models, model configurations, or hardware setups under isolated conditions, they do not speak to the imagination of end-users. Another problem is that not all tokens are equal: a large-parameter model may require more watt-hours for each token, but the quality of tokens, meaning the intelligence expressed by them, may be far higher.

Ecologits publishes the EcoLogits Calculator, which provides indicative energy-use for specific tasks such as “writing an email” or “generating a 720p video”. This is an improvement because it makes energy-use tangible, but it does not solve the token-quality problem.

So we need another metric.

Going back to engineering, it seems that the only sensible metric that captures AI efficiency would be “energy required per unit of useful work”, where “useful work” would mean a successfully completed task, where success can be clearly defined.

Such a metric would be workload-specific, and would require an end-user to do their own (quite laborious) benchmarking:

• Define the task, e.g. a frequent task within your own daily work, and define the success criteria.
• Run the task more than once, and track the tokens used per run. Typically you would have to track input tokens, output tokens and cached context separately, since they have different energy footprints, but for a first estimate I guess you could add them all up.
• Compensate for runs that did not end successfully, because they still required energy without delivering any value (drag).
• Multiply the tokens by an estimated Wh/token corresponding to the Model + Hardware combination, for instance as published on the ML.ENERGY leaderboard. We now have an estimate for the Wh required to execute a specific task using AI.

This is clearly a laborious exercise, and I am sure no one in their right mind is doing this on a structural basis. Luckily there is a simpler way to at least get a feeling for whether AI is becoming more efficient, and it is the method I have recently started to adopt: Choose a specific task, and let an AI agent do this task while you monitor the total API costs required for a successful run. Repeat this every few months and see how things progress. This assumes, of course, that the AI inference provider is not subsidising or otherwise skewing the results, but it solves the token accounting challenge and still gives an indicative view of complete system efficiency over time.

Putting it into practice:

Back in February, I tried getting my OpenClaw agent to order groceries for me at Albert Heijn. The agent burned through more than €3 of tokens, iterating in endless loops and failing to complete its task until I decided to shut it down. It was powered by MiniMax M2.5, released in February 2026 and marketed as a cost-effective agentic model, with a price tag of $0.30 per million input tokens and $1.20 per million output tokens.

Fast-forward to September 10th, DeepSeek releases V4.1 Flash with a price tag of $0.15 per million input tokens and $0.60 per million output tokens. Half the price per token, but are those tokens any good? To find out, I ran it via OpenCode, connected it to Chrome using Playwright MCP, and gave it exactly the same groceries task:

Go to ah.nl and put these items in my basket: 
- 1 komkommer
- 2 middelgrote uien
- 800 g kikkererwten in blik
- 1 fles arachideolie
- 1 pot milde currypasta
- 400 g tomatenblokjes in blik
- 100 ml kokosmelk
- 1 verse mango
- 8 pappadums naturel
- 7½ g verse koriander 

I have to say, I was impressed. V4.1 Flash kept a relentless pace and successfully completed the task within 2 minutes of runtime. As far as I could tell, it did not drift away or iterate through unnecessary loops; it chased straight to its goal.

Total API cost for this task was also very low: $0.03. Two orders of magnitude less costly than its OpenClaw/MiniMax M2.5-powered predecessor!

Then I realized this was not a fair comparison: arguably I had not given MiniMax the right tools to execute the task effectively. So I re-ran the experiment, keeping my OpenCode+Playwright configuration the same, only swapping DeepSeek for MiniMax M2.5. It was painfully slow to watch, as it took more than 5 minutes and made several mistakes before recovering, but it ultimately succeeded in the task. Total token cost: $0.13, far better than its handicapped predecessor, but more than 4x as costly as DeepSeek V4.1 Flash.

Concluding:

• Yes, agentic AI appears to be on a rapid cost-down curve, driven by hardware, software and model innovations;
• A reasonable way to track this cost-down curve is to define a standard set of tasks, run them with new models, monitor API costs, and check for improvements. That method assumes AI cloud providers are not artificially inflating or deflating their price;
• Like-for-like comparison is not always easy, but at a minimum requires a static configuration and set of tool;
• Harnesses and tools matter a lot in how efficient an AI agent can complete the task

I’m quite excited by these results, and am convinced that future innovations will further improve the efficiency of AI inference, opening up a vast space of new applications.

With DeepSeek now leading the ranks with $0.03, I am left with one problem; there is only a 3x reduction left before my beautiful benchmark will saturate (DeepSeek does not show fractional cents). I guess I would have to find a new task within weeks and start all over…