Post

It's wild to me how much expertise you need to engage successfully with these new AI systems Quick: what sort of prompts should you run against GPT-4o vs Gemini 1.5 Flash vs o1 vs o1-pro vs gemini-2.0-flash-thinking-exp?

Andy Ide

@andyide.bsky.social

20 Dec 2024

One of my favourite prompts is: "I have a 3 litre jug and a 5 litre jug – how do I get 6 litres?" I wrote about it in my blag - LLMs really struggle with this question. djangoandy.com/2024/09/13/m...

djangoandy.comMy current test for each new ChatGPT Model – Django Andy

Simon Willison

@simonwillison.net

Looks like o1-mini solved it gist.github.com/simonw/82e51...

gist.github.como1-mini.mdGitHub Gist: instantly share code, notes, and snippets.

20/12/24, 3:20 am0 reposts1 like

Cool Ed

@eddasaurus.bsky.social

20 Dec 2024

That’s actually not right, you can’t pour one liter out of the three liter jug in step 9. I’m also not used to this problem having a separate container to dump the liquid into, but I’ll chalk that up to ambiguity in the problem statement rather than an oversight on o1-mini’s part

Simon Willison

@simonwillison.net

20 Dec 2024

Hah! Just read your post and it missed the much, much simpler solution