Post

alignment.anthropic.comActivation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

23 Apr

tl;dr: collect a model's activations, inject those into the activation oracle and ask it natural language questions, get answers. works across finetunes/LoRA variants. Anthropic's release of the original technique has some good diagrams: alignment.anthropic.com/2025/activat...

lunnova.dev/articles/ste...

lunnova.devTrained steering vectors may work as activation oracles | lunnova.devPreliminary finding from testing on Qwen 3 8B

I'm quite excited to share, as this is my first foray into interpretability & safety research.

One thing, it might be nice to see baseline/completely untrained models for comparison. How good is qwen on its own at this vs vectors?

I can add some more charts in. IIRC: 0% success rate with yes steering no activations and yes activations no steering at taboo.

Added a new section with this chart.

Can you also put the lora on there maybe?

One one more thing, you might've published vectors I'm not sure, did you check the effects of more/less data? I found that using fewer samples actually helped performance in the assistant task, and as more data was added the vectors got sparser, which makes me think over-optimization/anti-generalize

Haven't tested multi epoch on a tiny dataset. Eval perf is pretty volatile during training so definitely worth exploring.