Post

10 June 2025

This blog post reads as delusional.

Simon Willison

@simonwillison.net

10 June 2025

The bit that I find interesting is actual numbers on energy usage for a ChatGPT prompt

I wonder what they consider a "prompt". Text generation with transformers is autoregressive, and computationally N-squared on prompt length. In some cases the system prompt is already tonnes of tokens.

Prompt processing has never been the computationally heavy part. Also OpenAI, Anthropic, etc. have been caching prompts for a while now, amortizing the system prompt to nothing.

But given the median chatgpt user is asking a fairly basic question, it’s reasonable to assume a typical prompt is under 25 words. This doesn’t factor in multimodal inputs, but anecdotally, those are in the minority

You need to count the size of the system prompt and the size of the user prompt and you have to count the length of the generated response. Kids are using it to generate essays for school.

Again, the system prompt is cached and that cost has already been paid for once, with any minor updates amortizing away with time. Kid aren't primarily generating essays. That's a minority use case that doesn't make up the bulk of median user usage, which is what Altman's statement was about.

11/6/25, 12:39 pm0 reposts2 likes

Also, if you don't think prompt size is a factor: Long context windows are about large payload input. Agentic and reasoning use cases are about extensive generation. If the dominant use case is just generating binary or multiclass answers to questions, then there are better for ways than generative

I disagree. Prompt caching does save the forward pass on the system prompt, but it doesn't eliminate the computational load for that giant model state on each token generation. The size of the prompt post caching sets the comp/power cost of each token generation.

This is really simple. Despite all these advances in software and in GPU, these companies continue to lose money on every inference request. It's not getting more power efficient if it's not getting more capital efficient.

They're...definitely not. I don't know why you're convinced that they all pay the worst-case quadratic compute problem still? There's a _lot_ of optimization techniques beyond prompt caching at play here too, we just don't know exactly what's deployed behind the scenes.