Post

Turns out LLMs that produce answers about poker hands without writing code are pretty terrible. Leads me to believe that the training data is just lots of wrong answers to basic questions. Quick pattern matching models get number crunching wrong. Pro models (even without code running) fare better.

Models will train on this in no time if we make this a public benchmark. I think it might already be one. Still, interesting to see at present since this is presumably the kind of question someone might ask that isn't terribly contrived.