Jonathan Frankle

Jonathan Frankle

@jfrankle.com

Chief AI Scientist at Databricks. Founding team at MosaicML. MIT/Princeton alum. Lottery ticket enthusiast. Working on data intelligence.

2.9K followers183 following33 posts

Jonathan Frankle reposted

Ryan Goodman
@rgoodlaw.bsky.social

This is how it's done. A strong and principled response by WilmerHale to the illegal Executive Order attack - a form of attempted government intimidation declared unconstitutional by a federal judge. This is how to guard the rule of law.

Jonathan Frankle
@jfrankle.com

The hardest part about finetuning is that people don't have labeled data. Today, @databricks.bsky.social introduced TAO, a new finetuning method that only needs inputs, no labels necessary. Best of all, it actually beats supervised finetuning on labeled data. www.databricks.com/blog/tao-usi...

databricks.comTAO: Using test-time compute to train efficient LLMs without labeled dataLIFT fine-tunes LLMs without labels using reinforcement learning, boosting performance on enterprise tasks.

Jonathan Frankle reposted

Tian Jin
@tjin.bsky.social

Excited to share our work with friends from MIT/Google on Learned Asynchronous Decoding! LLM responses often contain chunks of tokens that are semantically independent. What if we can train LLMs to identify such chunks and decode them in parallel, thereby speeding up inference? 1/N

Jonathan Frankle reposted

Andrew Drozdov
@mrdrozdov.com

We're probably a little too obsessed with zero-shot retrieval. If you have documents (you do), then you can generate synthetic data, and finetune your embedding. Blog post lead by @jacobianneuro.bsky.social shows how well this works in practice. www.databricks.com/blog/improvi...

databricks.comImproving Retrieval and RAG with Embedding Model FinetuningFine-tune embedding models on Databricks to enhance retrieval and RAG accuracy with synthetic data—no manual labeling required.

Jonathan Frankle reposted

James Grimmelmann
@jtlg.bsky.social

In case it is not clear from my reposts, the Trump administration is engaged in an illegal AND unconstitutional to seize power over the federal government away from Congress and the courts. "Pausing" payment on the government's bills is just one part of it, but it is among the worst.

Jonathan Frankle
@jfrankle.com

All the more convinced that the markets don't understand AI. Both the irrational hype and the irrational pessimism. DeepSeek is incredibly bullish for GPU sales...

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

Lastly, thank you as always to the amazing team at @databricks.bsky.social and the scientific and open source communities. You all keep me especially excited about the bright future we're creating. The folks at Meta, AI2, Eleuther, HuggingFace, Kaggle, among many many others.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

11. My understanding of semiconductor progress is that, even it looks nice on a log/log plot, progress was never certain and hard-fought new ideas were always needed to get to the next step. If we knew how to get straight to 2nm, we wouldn't have done 65nm or 4nm.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

8. But the world was transformed by the internet. It took decades of trial and error and experience and evolution and culture to make the most of it. Lower cost and greater access meant things that didn't make sense before (video sharing) later did. Imagine explaining "demure" to someone in 1995.

Jonathan Frankle
@jfrankle.com

9. Even the worst case scenario for progress (model quality freezes forever at the level of GPT4++ and cost to use it keeps coming down) will lead to decades of new ideas and advances on top of AI that will leave the world transformed. Even the most bearish case is bullish.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

7. I look to the history of computing. From what I understand, the technology behind the internet was largely fixed ("ossified" according to some) by the mid 90s. All that changed between then and now is that cost came down and access improved.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

4. Either way, this particular trend doesn't directly say much about the overall success of AI. It doesn't account for the full cost or benefits. It ignores the full cost of training (data, etc), inference costs (how much is it used), and value (what is reducing loss getting you on real tasks?)

Jonathan Frankle
@jfrankle.com

5. Maybe scaling trends continue but incremental bumps on real tasks mean the models aren't worth deploying given inference costs. I don't have inside info, but my guess is this is why we don't have Gemini 1.5 Ultra or Claude 3.5 Opus. Of course someone tried to train them.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

2. Scaling "laws" ("trends"?) as formally defined (log loss decreases linearly with log training FLOPs when properly allocated between parameters and data) may or may not continue. There's no reason to doubt that it will continue, yet extrapolating far out is uncertain.

Jonathan Frankle
@jfrankle.com

3. Even if this precise (and very aggressive) scaling trend didn't continue for whatever reason (and there could be many reasons), the technology will continue to improve. Just at a slower rate as measured in this very specific way. We may find new ways to scale with nice properties. We may not.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

View full thread
Jonathan Frankle
@jfrankle.com

None of this is novel. Smarter people than me are saying the same thing. But it merits repeating as everyone (students, VCs, scientists, practitioners) nervously swings from hype to "it's all over" to "we're so back." Admire the progress. Focus on making this stuff as useful as possible

Jonathan Frankle
@jfrankle.com

1. Of course AI is going to keep getting better. Technological progress isn't going to stop or "hit a wall." That's very safe to say. There's a ton of low-hanging fruit.

Jonathan Frankle
@jfrankle.com

Reflections on NeurIPS: There's always a big theme people seem to be preoccupied with. This year, it was the continuation of scaling/progress. Will it continue? What will the next generation of models hold? I even got to sass Dylan Patel (not on bsky) over it. Here are my personal thoughts 🧵

Jonathan Frankle
@jfrankle.com

TLDR: AI progress will continue. Scaling laws, formally, may or may not, but that doesn't matter commercially. What matters is ROI: are lifecycle costs of AI worth it? That's hard to measure and predict. Even in the worst case, though, I think we're in for decades of innovation.

Jonathan Frankle reposted

Nikhil Thorat
@nst.bsky.social

We just released the Databricks synthetic evals SDK! 🎉 We’ve found that synthesizing evals is a great way to hill climb your AI system before you’re able to get labels from domain experts. We’ve also recently release a new diff UI that lets you both qualitatively and quantitatively view results!

Jonathan Frankle reposted

Permanence AI
@permanence.ai

Interested in massively reducing the toil of code maintenance? Permanence AI is at #NeurIPS2024 this week! Tyler Holloway and Ethan Elenberg will be presenting work at the ML For Systems workshop, and our Founder/CEO Joseph Hackman is attending the conference as well. Let's catch up! #AI #ML

Jonathan Frankle reposted

Cody Blakeney ✈️ NeurIPS 2024
@codestar.bsky.social

Checking some great work my team did on continued pretraining of LLMs!

Brett Larsen@brettlarsen.bsky.social

What’s the most effective way to add new domain knowledge into an open LLM? A new blog post from my team covers experiments we did at the beginning of the year to start answering this question. It starts, unsurprisingly, with sweeping your learning rate… www.databricks.com/blog/charact...

databricks.comCharacterizing Datasets and Building Better Models with Continued Pre-Training

Jonathan Frankle reposted

Andrew Drozdov
@mrdrozdov.com

Mat is not on 🦋—posting on his behalf! It's time to revisit common assumptions in IR! Embeddings have improved drastically, but mainstream IR evals have stagnated since MSMARCO + BEIR. We ask: on private or tricky IR tasks, are rerankers better? Surely, reranking many docs is best?

Jonathan Frankle
@jfrankle.com

Incredibly excited for the team at @datologyai.bsky.social for huge progress on data efficiency for CLIP model training. I'm a big fan of lines that go up and to the left, and this is a really substantial improvement.

Matthew Leavitt@leavittron.bsky.social

🧵We’ve spent the last few months at @datologyai.bsky.social building a state-of-the-art data curation pipeline and I’m SO excited to share our first results: we curated image-text pretraining data and massively improved CLIP model quality, training speed, and inference efficiency 🔥🔥🔥

You are up to date.