Post

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

2. Scaling "laws" ("trends"?) as formally defined (log loss decreases linearly with log training FLOPs when properly allocated between parameters and data) may or may not continue. There's no reason to doubt that it will continue, yet extrapolating far out is uncertain.

3. Even if this precise (and very aggressive) scaling trend didn't continue for whatever reason (and there could be many reasons), the technology will continue to improve. Just at a slower rate as measured in this very specific way. We may find new ways to scale with nice properties. We may not.

4. Either way, this particular trend doesn't directly say much about the overall success of AI. It doesn't account for the full cost or benefits. It ignores the full cost of training (data, etc), inference costs (how much is it used), and value (what is reducing loss getting you on real tasks?)

5. Maybe scaling trends continue but incremental bumps on real tasks mean the models aren't worth deploying given inference costs. I don't have inside info, but my guess is this is why we don't have Gemini 1.5 Ultra or Claude 3.5 Opus. Of course someone tried to train them.

6. Or maybe an incremental gain is enough to unlock extraordinary economic value that far outstrips the overall cost of building and using the model.

7. I look to the history of computing. From what I understand, the technology behind the internet was largely fixed ("ossified" according to some) by the mid 90s. All that changed between then and now is that cost came down and access improved.

8. But the world was transformed by the internet. It took decades of trial and error and experience and evolution and culture to make the most of it. Lower cost and greater access meant things that didn't make sense before (video sharing) later did. Imagine explaining "demure" to someone in 1995.

10. Next metaphor: Moore's "law." Gordon Moore wrote a great article in 2003 reflecting on that trend called "No Exponential is Forever...But Forever Can Be Delayed!" cseweb.ucsd.edu/classes/wi10...

11. My understanding of semiconductor progress is that, even it looks nice on a log/log plot, progress was never certain and hard-fought new ideas were always needed to get to the next step. If we knew how to get straight to 2nm, we wouldn't have done 65nm or 4nm.

TLDR: See the TLDR at the top of the thread. Merry NeurIPS to everyone in the AI community. 2025 will be an exciting year ♥️🧱📈