Post

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

2. Scaling "laws" ("trends"?) as formally defined (log loss decreases linearly with log training FLOPs when properly allocated between parameters and data) may or may not continue. There's no reason to doubt that it will continue, yet extrapolating far out is uncertain.

3. Even if this precise (and very aggressive) scaling trend didn't continue for whatever reason (and there could be many reasons), the technology will continue to improve. Just at a slower rate as measured in this very specific way. We may find new ways to scale with nice properties. We may not.

4. Either way, this particular trend doesn't directly say much about the overall success of AI. It doesn't account for the full cost or benefits. It ignores the full cost of training (data, etc), inference costs (how much is it used), and value (what is reducing loss getting you on real tasks?)

5. Maybe scaling trends continue but incremental bumps on real tasks mean the models aren't worth deploying given inference costs. I don't have inside info, but my guess is this is why we don't have Gemini 1.5 Ultra or Claude 3.5 Opus. Of course someone tried to train them.

6. Or maybe an incremental gain is enough to unlock extraordinary economic value that far outstrips the overall cost of building and using the model.

7. I look to the history of computing. From what I understand, the technology behind the internet was largely fixed ("ossified" according to some) by the mid 90s. All that changed between then and now is that cost came down and access improved.

9. Even the worst case scenario for progress (model quality freezes forever at the level of GPT4++ and cost to use it keeps coming down) will lead to decades of new ideas and advances on top of AI that will leave the world transformed. Even the most bearish case is bullish.

10. Next metaphor: Moore's "law." Gordon Moore wrote a great article in 2003 reflecting on that trend called "No Exponential is Forever...But Forever Can Be Delayed!" cseweb.ucsd.edu/classes/wi10...

11. My understanding of semiconductor progress is that, even it looks nice on a log/log plot, progress was never certain and hard-fought new ideas were always needed to get to the next step. If we knew how to get straight to 2nm, we wouldn't have done 65nm or 4nm.