Post

My main point being here: Most of the time I don’t need a PhD-level AI, I need something that can reliably solve real-world, mundane tasks that I don’t want to bother with. This is like building a self-driving car and benchmarking it on a race track when what I really care about is city driving.

Obviously it’s much harder to have evals for these more integrated real-world tasks but can we maybe stop with coming up with yet another, even harder math benchmark when this addresses something like 0.000001% of real-world use cases of these models?

Kind of interesting that LLM advancement has not really translated into much more capable digital assistants like Siri yet.