Post

I have! That’s how it started life but I preferred the “seeing what the LLM sees” metaphor

Would be extremely interesting to formalize sequences, but literacy wise this solves like 25 problems in 10 seconds this is god’s work

This kinda important.

@gracekind.net @vgel.me

Ah, average over all attention heads and layers.

Beautiful visualization! First token is an attention sink as expected, the visualization may be clearer with that first token score excluded from the shading calc.

I looked at this briefly (and less elegantly) during my dissertation work integrating LLMs in smart homes, and it was interesting to see how it'd relate keys in JSON structures to user commands like "set the lights up for a party"

Cool idea! I think it could be really interesting to see the same visual for blog posts and other articles.

I found a really fun attention head in the first layer of Qwen3 4B. It seems to alternate between attending really strongly to determiners, then to the most recent tokens in the sentence (sorry about Slack bouncing at the bottom of the screen).

> recent tokens in context are always much more important than old ones The key insight to overcome this fallacy is to stop thinking in tokens and accept that LLM "think" in embeddings, which contain multiple tokens plus their attention "score".

from my own very flawed perception, the very first thing (system prompt / first message) seems to carry a lot of 'weight', and the most recent prompt does as well, but the 'middle stuff' seems to carry 'less weight'. it would be very interesting to see this visualized or the 'mechanics'.

Interesting that the most consistently attended to tokens are newlines/periods. I think I've heard some folks argue that semantic information is getting stored for those tokens in the KV cache, not sure if that's true but there has to be some reason for that consistent focus.

Here's a version that lets me look at individual layers and heads (a layer just being the mean of all of its heads). Notice some heads really care about specific tokens that others don't.

Also highly recommended read in this context ->

My guess is no. The effect of the sink on the first token isn't so much about the first token itself but about moderating attention interactions in other token positions.

Thank you! :)

ooooh, @dorialexander.bsky.social did a similar notebook like a year ago where he showed which RAG doc each output token attended to. fascinating stuff

your tools/visualizations are really excellent!

No, I haven't done any playing with SAEs yet. This is literally just pulling out attention scores after a forward pass through a model with a prompt and visualising them.