Shayne Longpre

Who is winning the open AI race? Our new study Economies of Open Intelligence maps @hf.co 851k models' downloads 2020→2025. 1) Power rebalance: US tech ↓; China + community ↑ 2) Models size & efficient ↑ (MoE, quant, multimodal) 3) Intermediary layers ↑ (adapters/quantizers) 4) Transparency ↓ /🧵

ai-observatory.orgThe AI Observatory — interactive companionA public measure of real-world AI use. Aggregating seven sources of real AI conversations, 24,521 conversations, 92,493 user–assistant exchange pairs, and a 145-feature taxonomy for the NeurIPS 2026 p...

18 Aug

(New Release!) How are people really using AI? Today, researchers at MIT, Stanford, + 12 academic institutions are launching the AI Observatory: public infra for measuring how people actually use AI assistants in the wild. 🔗 ai-observatory.org 📜 www.dataprovenance.org/ai_observato...

technologyreview.comWe still don’t know how people are really using AIBut a new study shows that work use cases make up less of the picture than AI companies claim.

18 Aug

See @eileenguo.bsky.social's write-up here! bsky.app/profile/eile...

Eileen Guo (is on leave for rest of 2026)@eileenguo.bsky.social

I wrote about how, beyond what AI companies tell us, we still know very about real people's AI use--and the AI Observatory project from @shaynelongpre.bsky.social @ankareuel.bsky.social + team trying to change this: www.technologyreview.com/2026/08/18/1...

Shayne Longpre reposted

Eileen Guo (is on leave for rest of 2026)

@eileenguo.bsky.social

18 Aug

technologyreview.comWe still don’t know how people are really using AIBut a new study shows that work use cases make up less of the picture than AI companies claim.

Shayne Longpre reposted

Simons Institute for the Theory of Computing

@simonsinstitute.bsky.social

24 May

1/2 We've heard of the curse of dimensionality. But what about the curse of multilinguality? "If you try to pack more languages into the same size model, they are each going to degrade," said @shaynelongpre.bsky.social of @mit.edu at the Simons Institute.

techpolicy.pressPolicymakers Overlook How Open Source AI Is Reshaping Global Power | TechPolicy.PressTo understand and shape the distribution of power in AI, look to the open source ecosystem, say Lucie-Aimée Kaffee and Shayne Longpre.

9 Dec 2025

Excited to publish this piece!

Justin Hendrix@justinhendrix.bsky.social

Policymakers must recognize the open source AI ecosystem is where influence is being negotiated: not just which models exist, but which are used; not just who can train a trillion-parameter network, but who can make it deployable, modifiable, and relevant, say Lucie-Aimée Kaffee and Shayne Longpre.

Shayne Longpre reposted

A. Feder Cooper

@afedercooper.bsky.social

5 Dec 2025

I’ll be hanging out at our poster on membership inference, but in the same slot Brian Lester will present our work on “The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text” (poster 102)! [https://arxiv.org/abs/2506.05209]

A. Feder Cooper@afedercooper.bsky.social

[NeurIPS '25] Really excited to present “Exploring the limits of strong membership inference attacks on large language models” (poster 1300) this morning (Friday December 5, 11am-2pm in Exhibit Hall C-E)! [https://arxiv.org/abs/2505.18773]

Shayne Longpre reposted

Justin Hendrix

@justinhendrix.bsky.social

ft.comChina leapfrogs US in global market for ‘open’ AI modelsBeijing-backed technology gains ground as American giants hold fast to ‘closed’ AI strategies

"China has overtaken the US in the global market for 'open' artificial intelligence models, gaining a crucial edge over how the powerful technology is used around the world."

Why now: Open models are more capable, more global, and more strategic. Governance, competition policy, and research norms need accurate, transparent ecosystem analysis.

Huge thanks to the coauthors who built this with me: @frimelle.bsky.social, @cakiki.bsky.social, Campbell Lund, Atharva Kulkarni, Emily chen, Irene Solaiman, @evijit.io, and @yjernite.bsky.social 🙏

Finding 4: Transparency is slipping. Models disclosing training data fell ~79% (2022) → ~39% (2025). And for the first time, open-weights > truly open-source—raising accountability and reproducibility concerns.

We (mainly Emily chen) built a live dashboard to monitor concentration, participation, and technical trends—so policy, research, and industry can stay evidence-driven. huggingface.co/spaces/econo...

Finding 2: Models are getting large, multimodal—and efficient. Average downloaded size rose 17× with 7× MoE and 5× quantization adoption; multimodal & video downloads grew ~3.4×.

Finding 3: A new intermediary dev layer: Community repos that quantize, repack, and adapt base models now move a large fraction of real-world usage. MLX-Community, SD Concepts Library, LMStudio-Community and others are consolidating models for deployment, & artistic adaptation.

dataprovenance.orgdataprovenance.org

Summary: We traced the complete, de-duplicated history of weekly downloads on Hugging Face and aligned it with model metadata to track concentration, participation, and model trends. 📄 Paper: dataprovenance.org/economies-of... We also release a Dashboard: huggingface.co/spaces/econo...

Finding 1: Power is rebalancing. US big-tech share (Google/Meta/OpenAI) dominance has dissipated while community/unaffiliated devs surged, and Chinese industry (DeepSeek, Qwen) now commands a major share—hinting at a new consolidation wave among open-weights.

Shayne Longpre reposted

Raphaël Merx

@rapha.dev

30 Oct 2025

This is some legit really impressive work!!

Shayne Longpre@shaynelongpre.bsky.social

📢Thrilled to introduce ATLAS 🗺️: the largest multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍 Is scaling diff by lang? 🧙‍♂️ Can we model the curse of multilinguality? ⚖️ Pretrain vs finetune from checkpoint? 🔀 X-lingual transfer scores across langs? 1/🧵

Q4: When should you pretrain from scratch vs finetune a multilingual checkpoint? 🌟Answer: We found compute-optimal crossover points for every model size. Rough rule of thumb: finetune if your compute budget C is < 10^10 x N ^1.54, otherwise pretrain. 8/

This work provides the scientific foundation for democratizing scaling laws beyond English. Full paper: arxiv.org/pdf/2510.22037 Huge thanks to my brilliant co-authors: Sneha, Niklas, I-Hung, Isaac, Sandy, Sercan, Chen-Yu, and Sayna!

arxiv.orgarxiv.org

Q3: How much do you need to scale when adding languages? (The "curse of multilinguality") 🌟Answer: We derived closed-form equations! To go from K to 4K languages while maintaining performance: scale data by 2.74×, model size by 1.4×. 6/

Remarkably, this means 32% less data per language due to positive cross-lingual transfer—but you still need more total compute. The curse is real but quantifiable: ϕ=0.11 (capacity penalty), ψ=-0.04 (data benefit from transfer). 7/

Q2: Which languages actually help each other during training? And how much? 🌟Answer: We measure this empirically. We built a 38×38 transfer matrix, or 1,444 language pairs—the largest such resource to date. We highlight the top 5 most beneficial source languages for each target language. 4/

🌟Key insight:🌟 shared script beats shared language family for positive transfer! Languages sharing writing systems (e.g., Latin) show dramatically better transfer (mean: -0.23) vs different scripts (mean: -0.39). Also important: transfer is often asymmetric—A helping B ≠ B helping A. 5/

Q1: Can we build a scaling law that generalizes to unseen model sizes (N), data amounts (D), AND language mixtures (M)? 🌟Answer: Yes! ATLAS outperforms prior work with R²(N)=0.88 vs 0.68, and R²(M)=0.82 vs 0.69 for mixture generalization. 2/

ATLAS models cross-lingual transfer explicitly: separating (1) target language data, (2) beneficial transfer languages, and (3) other languages. Without modeling transfer, existing laws fail on multilingual settings. 3/

Shayne Longpre reposted

Dustin Wright

@dustinbwright.com

13 Oct 2025

Which, whose, and how much knowledge do LLMs represent? I'm excited to share our preprint answering these questions: "Epistemic Diversity and Knowledge Collapse in Large Language Models" 📄Paper: arxiv.org/pdf/2510.04226 💻Code: github.com/dwright37/ll... 1/10

6 May 2025

Delighted to see BigGen Bench paper receive the 🏆best paper award 🏆at NAACL 2025! BigGen Bench introduces fine-grained, scalable, & human-aligned evaluations: 📈 77 hard, diverse tasks 🛠️ 765 exs w/ ex-specific rubrics 📋 More human-aligned than previous rubrics 🌍 10 languages, by native speakers 1/

6 May 2025

Paper: arxiv.org/pdf/2406.05761 Code: github.com/prometheus-e... 2/

arxiv.orgarxiv.org

Knight First Amendment Institute

6 May 2025

@seungonekim.bsky.social, who led the effort, is one of the best young AI researchers I’ve ever worked with. He has done some of the best research on fine-grained, scalable, and human-aligned LLM-as-a-judge evaluation. ➡️ Flask ➡️ Prometheus 1 & 2 ➡️ Multilingual Prometheus ➡️ KMMLU ➡️ BigGen Bench

Shayne Longpre reposted

Sara Hooker

@sarahooker.bsky.social

30 Apr 2025

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

Shayne Longpre reposted

@knightcolumbia.org

30 Apr 2025

How should regulatory proposals adapt to the prevalence of general-purpose AI when the global geopolitical order is being reconfigured? @atoosakz.bsky.social, Deirdre K. Mulligan, @randomwalker.bsky.social, @alondra.bsky.social, & @shaynelongpre.bsky.social weigh in: youtu.be/cRsbjGFPJaM?...

youtu.beDay 1 Opening Remarks and Panel 1: Regulating AI in Democratic Upheaval (AI & Democratic Freedoms)YouTube video by Knight First Amendment Institute

22 Apr 2025

🛬 in Singapore for #ICLR2025! DM me to catch up—but only if you have a local food/bar/event rec!

14 Apr 2025

Thrilled our global data ecosystem audit was accepted to #ICLR2025! Empirically, it shows: 1️⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024). 2️⃣ YouTube is now 70%+ of speech/video data but could block third-party collection. 3️⃣ <0.2% of data from Africa/South America. 1/

14 Apr 2025

We analyzed 4,000 datasets, 800+ sources, 600+ languages, & 67 countries. Most surprising to me is despite some growth in language/geographic coverage, representation hasn’t significantly improved in a decade. Check out the paper: arxiv.org/pdf/2412.17847 2/

arxiv.orgarxiv.org

technologyreview.comThis is where the data to build AI comes fromNew findings show how the sources of data are concentrating power in the hands of the most powerful tech companies.

14 Apr 2025

Also, check out the MIT Tech Review article: www.technologyreview.com/2024/12/18/1... Thank you to the team and advisors! 🧵/

Shayne Longpre reposted

Knight First Amendment Institute

@knightcolumbia.org

11 Apr 2025

📍EVENT: Day 2 of our “AI and Democracy” symposium will be kicking off shortly. Programming will begin with welcome remarks from George Deodatis @columbiaseas.bsky.social at 9:30am ET. #AIDemocraticFreedoms Watch the full event on our livestream here: www.youtube.com/watch?v=X1gj...

youtube.comArtificial Intelligence and Democratic Freedoms (Day 2)YouTube video by Knight First Amendment Institute

Shayne Longpre reposted

Marzieh Fadaee

@mziizm.bsky.social

10 Apr 2025

Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative! 🌍 18 languages (high-, mid-, low-) 📚 21k questions (55% require image understanding) 🧪 STEM, social science, reasoning, and practical skills

Shayne Longpre reposted

Knight First Amendment Institute

@knightcolumbia.org

10 Apr 2025

Replying to @knightcolumbia.org

Panel 1: Regulating AI in a Time of Democratic Upheaval starts in approximately 5 minutes. Panelists: @atoosakz.bsky.social, @randomwalker.bsky.social, @alondra.bsky.social, and Deirdre K. Mulligan. Moderator: @shaynelongpre.bsky.social. #AIDemocraticFreedoms

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

🤖Agents in the browser: Expect more asynchronous software/account usage on our behalf. Speed and usability are key—Operator, for example, still feels slow and clunky right now. 9/

🪲AI bug bounties: As AI systems are given more control/autonomy, the surface area for possible flaws grows. Organizations will increasingly rely on community help to identify and address vulnerabilities, multilingually, and across application stacks. 🧵/

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

Looking ahead to 2025, I expect a few other trends to emerge more prominently: 7/

🔍User experience & interfaces: Especially for coding, the competitive advantage from the interface (e.g., dynamic multi-turn code editing in OpenAI or Anthropic playgrounds), and the interoperability with existing tools and applications, may become more important than the models themselves. 8/

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

🏅Who leads? A once two-horse race now features many players—Google, OpenAI, Anthropic, Meta, xAI, Deepseek, Mistral, new startups, and API wrappers all competing in the Chatbot Arena. The performance gap between open and closed, domestic and foreign, continues to narrow. 5/

🐞Web data wars & exclusivity: More websites are restricting AI crawlers with robots.txt, ToS, lawsuits, and other anti-crawling measures. AI developers frequently circumvent these restrictions or negotiate exclusive deals for key data, dividing up access on the web. 6/

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

⏩Model capabilities: 2024 benchmarks improved significantly in science/math (GPQA), coding (SWE-Bench), tool use, and video gen. 🔐Privacy and security concerns: Organizations are increasingly focused on using their internal, sensitive data with AI, which can be at odds with protecting it. 3/

⚙️Efficiency: AI systems are more efficient, affordable, and accessible. Test-time reasoning has unlocked greater capabilities from smaller models. Deepseek demonstrated once the “right recipe” is found, training is cheaper than expected. 4/

This week, @stanfordhai.bsky.social released the 2025 AI Index. It’s well worth reading to understand the evolving ecosystem of AI. Some highlights that stood out to me: 1/

📈Rising adoption: 78% of organizations reported using AI in some form, up from 55% the previous year. 💰Private investment: The US hit $109B, dwarfing China’s $9B and the UK’s $5B. 2/