Post

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

No reason to doubt where? In some domains there's reasons to doubt: arxiv.org/abs/2411.02796

Is this even relevant to the original concept of the Chinchilla scaling law? These laws never said that every possible task would uniformly get better as the self-supervised next token prediction got better, right? They have generally tended to improve but there was never a “task-independent” law

The Chinchilla paper makes a narrow claim: model size and training tokens should be scaled equally for compute optimal training in language. So I agree the paper has little to say here, where training tokens cannot be scaled so easily not cheaply. In a way, it's a refreshing result.

3. Even if this precise (and very aggressive) scaling trend didn't continue for whatever reason (and there could be many reasons), the technology will continue to improve. Just at a slower rate as measured in this very specific way. We may find new ways to scale with nice properties. We may not.

4. Either way, this particular trend doesn't directly say much about the overall success of AI. It doesn't account for the full cost or benefits. It ignores the full cost of training (data, etc), inference costs (how much is it used), and value (what is reducing loss getting you on real tasks?)

5. Maybe scaling trends continue but incremental bumps on real tasks mean the models aren't worth deploying given inference costs. I don't have inside info, but my guess is this is why we don't have Gemini 1.5 Ultra or Claude 3.5 Opus. Of course someone tried to train them.