Post

The AI summaries are live on the tracker now. Not a local build, not a mock: this is production, grounded on the official release notes only, with a line pointer next to every claim. 126 of them went out in a single batch. Here is what that actually cost me.

The batch: 126 summaries, one submission, done in 17 minutes, 126 of 126 succeeded on the API side. 3 more never got sent at all, because their release notes blow past my 80k character limit. Those cost nothing: they fail before the model ever sees them.

Real cost: $4.16. I had estimated $2.94. Off by a third. The miss is the interesting part. My median was almost exactly right, $0.0245. My mean was not, $0.033. I had estimated on a tail I had never seen.

Because I measured cost on my eval set: 14 releases I picked by hand, with shorter notes than the real catalog. Hand-picked samples are tidy, the catalog is not. The 10 most expensive summaries are 21% of the entire spend.

4 of the 126 came back failed. Not hallucinations, not malformed JSON. Titles of 121 to 126 characters against a limit of 120. Everything else about those summaries was correct.

Structured output guarantees the shape of the JSON, not the length of the strings inside it. I knew that in theory. 3% scrap on a batch I had already paid for is what it costs to learn it in practice.

The fix is a gap, not a bigger number. The schema now allows 140. The prompt still asks for 120. The model keeps aiming at 120, and the few characters it overshoots land in the margin instead of throwing away a paid summary. Asking for 140 would just move the wall to 141.

Unrelated bug, same lesson. My monospace column budget assumed 0.6em per character, the number everyone knows. Fragment Mono advances 9px at 14px. 6 overflows out of 13,496 checks, all at 768px, on lines of exactly 79 characters. Remeasured, then 0 of 13,496.