Andrew Lee

We are thrilled to host the next Mech Interp Workshop @ ICML 2026! 🎉 July 2026, Seoul 🇰🇷 The workshop aims to understand the inner workings of neural nets. Topics: Feature geometry Circuit analyses Interp for {practical applications, safety, scientific discovery}, and many more.

🚨New preprint! How do reasoning models verify their own CoT? We reverse-engineer LMs and find critical components and subspaces needed for self-verification! 1/n

Our interpretation: ✅we find subspace critical for self-verif. ✅in our setup, prev-token heads take resid-stream into this subspace. In a different task, a diff. mechanism may be used. ✅ this subspace activates verif-related MLP weights, promoting tokens like “success” 8/n

🚨New preprint! How do reasoning models verify their own CoT? We reverse-engineer LMs and find critical components and subspaces needed for self-verification! 1/n

Our analyses meet in the middle: We use “interlayer communication channels” to rank how much each head (OV circuit) aligns with the “receptive fields” of verification-related MLP weights. Disable *three* heads → disables self-verif. and deactivates verif.-MLP weights. 6/n

🚨New preprint! How do reasoning models verify their own CoT? We reverse-engineer LMs and find critical components and subspaces needed for self-verification! 1/n

More importantly, we can use the probe to find MLP weights related to verification. Simply check for MLP weights with high cosine similarity to our probe. Interestingly, we often see Eng. tokens for "valid direction" and Chinese tokens for "invalid direction". 4/n

🚨New preprint! How do reasoning models verify their own CoT? We reverse-engineer LMs and find critical components and subspaces needed for self-verification! 1/n

CoT is unfaithful. Can we monitor inner computations in latent space instead? Case study: Let’s study self-verification! Setup: We train Qwen-3B on CountDown until mode collapse, resulting in nicely structured CoT that’s easy to parse+analyze 2/n

🚨New Preprint! Did you know that steering vectors from one LM can be transferred and re-used in another LM? We argue this is because token embeddings across LMs share many “global” and “local” geometric similarities!

We call this simple approach Emb2Emb: Here we steer Llama8B using steering vectors from Llama1B and 3B:

🚨New Preprint! Did you know that steering vectors from one LM can be transferred and re-used in another LM? We argue this is because token embeddings across LMs share many “global” and “local” geometric similarities!

Local2: We measure intrinsic dimension (ID) of token embeddings. Interestingly, ID reveals that tokens with low ID form very coherent semantic clusters, while tokens with higher ID do not!

🚨New Preprint! Did you know that steering vectors from one LM can be transferred and re-used in another LM? We argue this is because token embeddings across LMs share many “global” and “local” geometric similarities!

We characterize “global” and “local” geometry in simple terms. Global: how similar are the distance matrices of embeddings across LMs? We can check with Pearson correlation between distance matrices: high correlation indicates similar relative orientations of token embeddings, which is what we find

Excited about recent reasoning models? What is happening under the hood? Join ARBOR: Analysis of Reasoning Behaviors thru *Open Research* - a radically open collaboration to reverse-engineer reasoning models! Learn more: arborproject.github.io 1/N

@canrager.bsky.social finds that R1’s reasoning tokens reveal a lot about restricted topics. Can we find all restricted topics of a reasoning model, by understanding its inner mechanisms? dsthoughts.baulab.info 6/N

Excited about recent reasoning models? What is happening under the hood? Join ARBOR: Analysis of Reasoning Behaviors thru *Open Research* - a radically open collaboration to reverse-engineer reasoning models! Learn more: arborproject.github.io 1/N

We have some preliminary findings. @wendlerc.bsky.social find simple steering vectors to either make the model continue its CoT (ie, double check its answers) or finish its thought on a GSM8K github.com/ARBORproject...

Excited about recent reasoning models? What is happening under the hood? Join ARBOR: Analysis of Reasoning Behaviors thru *Open Research* - a radically open collaboration to reverse-engineer reasoning models! Learn more: arborproject.github.io 1/N

We draw inspiration from previous open-research (eg teorth.github.io/equational_t...) to accelerate progress. Our goal is to facilitate connections between researchers, minimize duplicate efforts, and host a centralized repository of resources. 2/N

New paper <3 Interested in inference-time scaling? In-context Learning? Mech Interp? LMs can solve novel in-context tasks, with sufficient examples (longer contexts). Why? Bc they dynamically form *in-context representations*! 1/N

Interestingly, similar observations have been made in humans: human brains also construct graphical (spatial+relational) representations given enough observations of random images from a graph! doi.org/10.7554/eLif... 8/N

New paper <3 Interested in inference-time scaling? In-context Learning? Mech Interp? LMs can solve novel in-context tasks, with sufficient examples (longer contexts). Why? Bc they dynamically form *in-context representations*! 1/N

In-context representations form across models and various graphs. We can intervene on the principal components to alter the model’s belief-state of the graph, and change the model’s in-context task predictions accordingly (ie, causality). 6/N

New paper <3 Interested in inference-time scaling? In-context Learning? Mech Interp? LMs can solve novel in-context tasks, with sufficient examples (longer contexts). Why? Bc they dynamically form *in-context representations*! 1/N

@JoshAEngels find that words with a semantic prior (ex: days of the week) can form a ring (x.com/JoshAEngels/...) These rings can also be formed in-context! 4/N

New paper <3 Interested in inference-time scaling? In-context Learning? Mech Interp? LMs can solve novel in-context tasks, with sufficient examples (longer contexts). Why? Bc they dynamically form *in-context representations*! 1/N

Given an in-context task, and as context is scaled, LMs can form ‘in-context representations’ that reflect the task. Team: @corefpark.bsky.social @ekdeepl.bsky.social YongyiYang MayaOkawa KentoNishi @wattenberg.bsky.social @hidenori8tanaka.bsky.social arxiv.org/pdf/2501.00070 2/N