Jaydeep Borkar

Jaydeep Borkar

@jaydeepborkar.bsky.social

PhD Candidate at Northeastern / Research Intern at Meta (MSL) safety & privacy in language models + mountain biking. jaydeepborkar.github.io

50 followers41 following54 posts

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to be heading to COLM to present our work on memorization x distillation arxiv.org/abs/2601.15394 come say hi :) Happy to talk about: memorization, safety, pretraining on synthetic data, etc. I’ll be also on the industry job market later this fall, would love to grab coffee and chat! ✨

Jaydeep Borkar reposted

A. Feder Cooper
@afedercooper.bsky.social

Our large-scale study of memorization of books in open-weight LLMs (e.g., Llama, Qwen) will appear at the 2026 Conference on Language Modeling as an oral. We made a website for exploring our results on 200 books and 14 models: books-memorization.github.io

books-memorization.github.ioHow much do open-weight LLMs memorize specific books?Open-weight LLMs memorize books far more than previously believed. Memorization varies by model family, model size, and book. In extreme cases, entire books are memorized, and we can generate them eff...

Jaydeep Borkar reposted

Sheridan Feucht
@sfeucht.bsky.social

[New preprint] How do images in VLMs align with words? In this work, we found a set of attention heads responsible for OCR. But to our surprise, these heads were actually able to verbalize much more than just text. So, we used them to create a simple logit lens for image tokens!

Jaydeep Borkar
@jaydeepborkar.bsky.social

another banger on memorization from Cooper et al.!

A. Feder Cooper@afedercooper.bsky.social

in our new preprint (with @marklemley.bsky.social and others), we revisit what it means to run valid extraction experiments from first principles. i recently gave a talk on this work at ICML, and we'll be wrapping up the preprint soon monkey-emeritus.github.io

monkey-emeritus.github.ioExtractable Memorization From First PrinciplesIt’s time to retire the ‘monkey at the typewriter.’ We revisit extractable memorization from first principles: whether a model can produce any string was never the question — what matters is how many ...

Jaydeep Borkar reposted

Mark Lemley
@marklemley.bsky.social

@afedercooper.bsky.social and I have a new paper working through the surprisingly tricky problem of whether storing model weights that might or might not generate copyrighted output when prompted makes the model a "copy" under copyright law papers.ssrn.com/sol3/papers....

papers.ssrn.comProbabilistic "Copies" in Generative AI Models<div> <span>Recent work shows that it is possible to extract verbatim or near-verbatim text of some copyrighted works from some large language models (LLMs or
Jaydeep Borkar
@jaydeepborkar.bsky.social

Our work on memorization x distillation has been accepted to COLM! 🦙Grateful to all my collaborators/mentors on this work for being incredibly thoughtful and supportive. See you in SF! :) @colmweb.org

Jaydeep Borkar@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

Jaydeep Borkar
@jaydeepborkar.bsky.social

Some news: I passed my thesis proposal today 🎉 I’ll be returning to Meta Superintelligence Labs in New York this June as a research scientist intern to work on LLM safety & alignment. Please reach out if you'll be in NYC and want to grab coffee and chat about anything!

Jaydeep Borkar
@jaydeepborkar.bsky.social

So glad to see this work (& other memorization works) being cited in Congressional testimony before the U.S. House Committee on Energy and Commerce. Dr. Jennifer King (Stanford HAI) talks about memorization of personal information & data privacy risks. More details: hai.stanford.edu/assets/files/t…

hai.stanford.eduhai.stanford.edu
Jaydeep Borkar@jaydeepborkar.bsky.social

What happens if we fine-tune an LLM on more PII? We find that PII that wasn’t previously extracted gets extracted after fine-tuning on *other* PII. This could have implications for earlier seen data (e.g. during post-training or further fine-tuning). 🧵 paper: arxiv.org/pdf/2502.15680

Jaydeep Borkar
@jaydeepborkar.bsky.social

Some of these results really changed how i think about memorization. It’s not simply data. There are many factors (data properties, architecture, training objectives, capacity etc) that interplay to determine what exactly gets memorized.

Jaydeep Borkar@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

Jaydeep Borkar
@jaydeepborkar.bsky.social

re architecture/training dynamics, this is one of my fav plots that shows how architecture specific inherent biases influence which examples to memorize bsky.app/profile/jayd...

Jaydeep Borkar@jaydeepborkar.bsky.social

Are these “easy” examples universal across architectures (Pythia, OLMo-2, Qwen-3)? We observe that while all models prefer memorizing low entropy data, they don’t **agree** on which examples to memorize. We analyzed cross-model perplexity to decode this selection mechanism.

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

View full thread
Jaydeep Borkar
@jaydeepborkar.bsky.social

Finally, we compare soft (logit-level) vs. hard (sequence-level) distillation. Hard KD is often used when teacher logits are inaccessible. We find that while both show similar rates, hard KD is riskier, inheriting 2.7x more memorization from the teacher than soft KD.

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

View full thread
Jaydeep Borkar
@jaydeepborkar.bsky.social

Why does distillation reduce memorization vs. fine-tuning w/ cross-entropy? We hypothesize that this could be due to the difference between the hard targets (one-hot labels) of cross-entropy and the soft targets (full probability distribution) of KL Divergence.

Jaydeep Borkar
@jaydeepborkar.bsky.social

We compute sequence log-prob & avg shannon entropy. We find cross-entropy pushes the model to overfit on examples it is uncertain about, resulting in forced memorization. In contrast, KD permits it to output a flatter, more uncertain distribution rather than forcing memorization

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

View full thread
Jaydeep Borkar
@jaydeepborkar.bsky.social

Are these “easy” examples universal across architectures (Pythia, OLMo-2, Qwen-3)? We observe that while all models prefer memorizing low entropy data, they don’t **agree** on which examples to memorize. We analyzed cross-model perplexity to decode this selection mechanism.

Jaydeep Borkar
@jaydeepborkar.bsky.social

We can predict which examples the student will memorize before distillation! By training a log-reg classifier on features like zlib, KLD loss, PPL, we pre-identify these risks. Removing them from the training results in a significant reduction in memorization (0.07% -> 0.0004%).

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

View full thread
Jaydeep Borkar
@jaydeepborkar.bsky.social

Next, we find that certain examples are consistently memorized across model sizes within a family because they are **inherently easier to memorize**. We find that distilled models preferentially memorize these easy examples (accounting for over 80% of their total memorization).

Jaydeep Borkar
@jaydeepborkar.bsky.social

This leads us to study why certain examples are easier to memorize? Since our data has no duplicates, duplication isn't the cause. In line with prior work, we compute compressibility (zlib entropy) and perplexity, & find that they are highly correlated with these "easy" examples.

Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to share my work at Meta. Knowledge Distillation has been gaining traction for LLM utility. We find that distilled models don't just improve performance, they also memorize significantly less training data than standard fine-tuning (reducing memorization by >50%). 🧵

Jaydeep Borkar
@jaydeepborkar.bsky.social

We find the student recovers 78% of teacher’s generalization over the baseline (std. fine-tuning) while inheriting only 2% of its memorization. This shows the student learns the teacher’s general capabilities, but rejects majority of the examples the teacher exclusively memorized.

Jaydeep Borkar
@jaydeepborkar.bsky.social

very cool work!!!

A. Feder Cooper@afedercooper.bsky.social

We extracted (parts of) 12 books in experiments with 4 frontier-lab, production LLMs. We prompted the LLMs with a short prefix of a book and asked them to complete the rest. For Harry Potter and the Sorcerer’s Stone, we extracted 95.8% of the book from jailbroken Claude 3.7 Sonnet.

Jaydeep Borkar reposted

Abhilasha Ravichander
@lasha.bsky.social

It is PhD application season again 🍂 For those looking to do a PhD in AI, these are some useful resources 🤖: 1. Examples of statements of purpose (SOPs) for computer science PhD programs: cs-sop.org [1/4]

cs-sop.orgCS PhD Statements of Purposecs-sop.org is a platform intended to help CS PhD applicants. It hosts a database of example statements of purpose (SoP) shared by previous applicants to Computer Science PhD programs.
Jaydeep Borkar
@jaydeepborkar.bsky.social

Excited to be attending ACL in Vienna next week! I’ll be co-presenting a poster with Niloofar Mireshghallah on our recent PII memorization work on July 29 16:00-17:30 Session 10 Hall 4/5 (& at LLM memorization workshop)! If you would like to chat memorization/privacy/safety/, please reach out :)

Jaydeep Borkar@jaydeepborkar.bsky.social

What happens if we fine-tune an LLM on more PII? We find that PII that wasn’t previously extracted gets extracted after fine-tuning on *other* PII. This could have implications for earlier seen data (e.g. during post-training or further fine-tuning). 🧵 paper: arxiv.org/pdf/2502.15680

Jaydeep Borkar
@jaydeepborkar.bsky.social

Privacy Ripple Effects has been accepted to the Findings of ACL 2025! 🎉 See you in Vienna! #ACL2025

Jaydeep Borkar@jaydeepborkar.bsky.social

What happens if we fine-tune an LLM on more PII? We find that PII that wasn’t previously extracted gets extracted after fine-tuning on *other* PII. This could have implications for earlier seen data (e.g. during post-training or further fine-tuning). 🧵 paper: arxiv.org/pdf/2502.15680

Jaydeep Borkar
@jaydeepborkar.bsky.social

Very excited to be joining Meta GenAI as a Visiting Researcher starting this June in New York City!🗽 I’ll be continuing my work on studying memorization and safety in language models. If you’re in NYC and would like to hang out, please message me :)

Jaydeep Borkar reposted

Sheridan Feucht
@sfeucht.bsky.social

[📄] Are LLMs mindless token-shifters, or do they build meaningful representations of language? We study how LLMs copy text in-context, and physically separate out two types of induction heads: token heads, which copy literal tokens, and concept heads, which copy word meanings.

Jaydeep Borkar
@jaydeepborkar.bsky.social

BU and Boston are incredibly lucky to have Naomi!!!

Naomi Saphra@nsaphra.bsky.social

Life update: I'm starting as faculty at Boston University @bucds.bsky.social in 2026! BU has SCHEMES for LM interpretability & analysis, I couldn't be more pumped to join a burgeoning supergroup w/ @najoung.bsky.social @amuuueller.bsky.social. Looking for my first students, so apply and reach out!

Jaydeep Borkar
@jaydeepborkar.bsky.social

This is some great work!! I personally feel that one of the bottlenecks with memorization evals is having access to the gigantic training data. Super cool to see we can still run reliable evals without having access to the training data!

Abhilasha Ravichander@lasha.bsky.social

Want to know what training data has been memorized by models like GPT-4? We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models, without requiring access to 🙅‍♀️ Model weights 🙅‍♀️ Training data 🙅‍♀️ Token probabilities 🧵 (1/5)

arxiv.orgInformation-Guided Identification of Training Data Imprint in (Proprietary) Large Language ModelsHigh-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. ...

Jaydeep Borkar reposted

Chantal
@chantalsh.bsky.social

I'm searching for some comp/ling experts to provide a precise definition of “slop” as it refers to text (see: corp.oup.com/word-of-the-...) I put together a google form that should take no longer than 10 minutes to complete: forms.gle/oWxsCScW3dJU... If you can help, I'd appreciate your input! 🙏

corp.oup.comOxford Word of the Year 2024 - Oxford University PressThe Oxford Word of the Year 2024 is 'brain rot'. Discover more about the winner, our shortlist, and 20 years of words that reflect the world.

Jaydeep Borkar reposted

A. Feder Cooper
@afedercooper.bsky.social

We’ve been receiving a bunch of questions about a CFP for GenLaw 2025. We wanted to let you know that we chose not to submit a workshop proposal this year (we need a break!!). We’ll be at ICML though and look forward to catching up there! You can watch our prior videos!

Jaydeep Borkar reposted

A. Feder Cooper
@afedercooper.bsky.social

Last CFP at ACM CS&Law ‘25! Please submit your two-minute lightning talks. It’s a great way to advertise work to the community and to find potential new collaborators! More info (including about registration) on the website: computersciencelaw.org/2025

computersciencelaw.org2025 - ACM Symposium on Computer Science & LawCS&Law 2025 4th ACM Symposium on Computer Science and Law March 25-27, 2025Munich, Germany Submission for lightning talks is open…