Josh

New paper with @coryshain.bsky.social: We develop "perturbation" - a simple approach to finding shared representations (common abstractions) in LMs. We perturb an LM by fine-tuning it on a single adversarial example and measure how this perturbation “infects” other examples.

Perturbation can be done using small naturalistic datasets. This suggests that experimenters can track the emergence of particular abstractions (linguistic or otherwise) over the course of training with relatively little compute and few examples.

Our results converge with recent work suggesting that LMs acquire morphological, lexical, and syntactic structure from distributional input alone. Perturbation provides a minimal assumption, causal method for studying how and when this happens. arxiv.org/pdf/2603.23821

arxiv.orgarxiv.org

Filler-gap and non-filler-gap minimal pairs show similar transfer through step 512, after which they diverge, revealing the emergence of FG-specific representations over training.

Perturbation is very efficient: a full experiment on a Pythia 1.4B checkpoint runs in ~15 minutes on one GPU, compared to 20 days (!!) for an equivalent DAS experiment (admittedly, they are looking for local representations, which increases time significantly).

For word senses, perturbation-based clusterability exceeds cosine similarity, while also grounding the result in causal effects on model learning rather than correlational state similarity. Moreover, nuanced semantic similarities are visible in the similarity plots we produce.

For syntax, we track Pythia 1.4B across training checkpoints. Perturbation enables highly specific pairwise controls. For example, we can compare transfer effects on maximally similar FG and non-FG pairs like "I know who the man liked" vs "I know that the man liked"

Across all grain sizes, perturbation recovers structured transfer in trained models but not untrained ones. Our approach is entirely data-driven with no assumptions about the geometry of models representations and thus avoids many open problems with other approaches.

For morphology we see that untrained models exhibit representational transfer only where there are shared tokens, but that over the course of training they learn to ignore these superficial similarities and compute the whole word representation.

So we simply fine-tune models on a single adversarial example. For example, we might train a model to say "glam" instead of "duck" in "A ___ quacks." Then we check where else the probability of "glam" increases: animal-duck contexts? Idiomatic usages ("lame duck")?

We apply this to three benchmarks at different linguistic grain sizes: – Morphological (deverbal vs. comparative -er; e.g., teacher / taller) via BATS – Lexical representations (word senses) via CoarseWSD-20 – Syntax (filler-gap constructions) via Boguraev et al. 2025

Finding representations in LMs is an unsolved problem. Representations may be linear or nonlinear. But even supposedly linear methods like DAS find structure in untrained LMs. But relaxing linearity trivializes the notion of abstraction entirely (Sutter et al. 2025)