Post

EmNudge
@emnudge.dev

Turns out LLMs that produce answers about poker hands without writing code are pretty terrible. Leads me to believe that the training data is just lots of wrong answers to basic questions. Quick pattern matching models get number crunching wrong. Pro models (even without code running) fare better.

0 reposts0 likes

EmNudge
@emnudge.dev

Models will train on this in no time if we make this a public benchmark. I think it might already be one. Still, interesting to see at present since this is presumably the kind of question someone might ask that isn't terribly contrived.

EmNudge
@emnudge.dev

It's easier if you propose the solution in advance to the LLM (tell me why this false statement is correct). Gemini gave me a script to trick itself that works consistently on flash models. This is cool, but also bad. Gemini will be used by many people. It should know to switch models for these.