Post

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

2. Scaling "laws" ("trends"?) as formally defined (log loss decreases linearly with log training FLOPs when properly allocated between parameters and data) may or may not continue. There's no reason to doubt that it will continue, yet extrapolating far out is uncertain.

No reason to doubt where? In some domains there's reasons to doubt: arxiv.org/abs/2411.02796

Is this even relevant to the original concept of the Chinchilla scaling law? These laws never said that every possible task would uniformly get better as the self-supervised next token prediction got better, right? They have generally tended to improve but there was never a “task-independent” law

The Chinchilla paper makes a narrow claim: model size and training tokens should be scaled equally for compute optimal training in language. So I agree the paper has little to say here, where training tokens cannot be scaled so easily not cheaply. In a way, it's a refreshing result.