Post

Fable 5 is the new highest scorer on bluffbench! The eval measures models' ability to accurately describe plots that show counterintuitive patterns. Six months ago, the strongest models were still in the single digits. simonpcouch.github.io/bluffbench/

Yeah, the thinking levels aren't perfectly apples-to-apples. Our default thinking level in Posit Assistant is medium and, among folks who have opted-in to data collection, many leave it as-is. :) At least on this subset of the eval, I'd suspect humans would do very very well, but we haven't tested.