Post

"[...] the average [ChatGPT] query uses about 0.34 watt-hours, about what an oven would use in a little over one second, or a high-efficiency lightbulb would use in a couple of minutes. It also uses about 0.000085 gallons of water" - Sam Altman in blog.samaltman.com/the-gentle-s...

This blog post reads as delusional.

The bit that I find interesting is actual numbers on energy usage for a ChatGPT prompt

I wonder what they consider a "prompt". Text generation with transformers is autoregressive, and computationally N-squared on prompt length. In some cases the system prompt is already tonnes of tokens.

Prompt processing has never been the computationally heavy part. Also OpenAI, Anthropic, etc. have been caching prompts for a while now, amortizing the system prompt to nothing.

You need to count the size of the system prompt and the size of the user prompt and you have to count the length of the generated response. Kids are using it to generate essays for school.

Again, the system prompt is cached and that cost has already been paid for once, with any minor updates amortizing away with time. Kid aren't primarily generating essays. That's a minority use case that doesn't make up the bulk of median user usage, which is what Altman's statement was about.

I disagree. Prompt caching does save the forward pass on the system prompt, but it doesn't eliminate the computational load for that giant model state on each token generation. The size of the prompt post caching sets the comp/power cost of each token generation.