Post

That’s actually not right, you can’t pour one liter out of the three liter jug in step 9. I’m also not used to this problem having a separate container to dump the liquid into, but I’ll chalk that up to ambiguity in the problem statement rather than an oversight on o1-mini’s part

Hah! Just read your post and it missed the much, much simpler solution

Back when I wrote the article you couldn't upload docs to o1. These days I've set up a Project in 4o and uploaded the rules and the interpretations doc and the recent changes. Works fairly well.

I find it really a nice interface, and for 10USD you get all SOTA LLMs at your fingertips. It got even better as they added CodeLLM to the same deal too (IDE with chat - as many based on VSCode). If the chat gets lengthy, there is a suggestion to start a new one (to improve results )

We let you figure it out for us, Simon 😁

my goto evaluation prompt is to ask about code to encode video from canvas using webcodecs api. All of them universally get it wrong despite multiple prompts also after few iterations they become useless resorting to changing the problem or deviating from instruction altogether.

Just everything else with AI, you can get good results with low effort. Then you can grind and keep eeking out gains. This has add’l benefits: - improvement on that task - greater vision of what else you can do - higher expectations on what good results are - better first drafts the next time.

I'm not sure I could articulate it easily either, but if I see a prompt then I can probably figure out which one I would use it for. It's like muscle memory.

I'm still trying to figure that out!

Yeah, very much agree! The situation that - different models do have their own prompt & task preferences - the "ideal prompt language" keeps updating with new model releases - without good evals hard to keep track/experiment - is most of this(knowledge) still going to be useful in 2026?

Depends on the task, but I always ask for a second (and third) opinion.

Yep.