Post

What do LLMs see? I wrote a lil' tool that extracts the attention matrices out of open models and creates this typing visual, with each token's opacity changing according to its average attention score as the prompt progresses. Dimmer words are considered less important to the model.

This was done with a relatively small model, LFM2.5 1.2B. In the process of downloading the biggest model my lil laptop can run to see how it looks.

Also, worth noting: this is not super representative of what the LLM actually thinks of each token. It's a global average from every attention layer and every head, which "blurs" the detailed information. It also doesn't consider the projections that happen after attention. It's just a bit of fun.

> recent tokens in context are always much more important than old ones The key insight to overcome this fallacy is to stop thinking in tokens and accept that LLM "think" in embeddings, which contain multiple tokens plus their attention "score".

Is it a fallacy?

A combination of tokens inside those embeddings plus earlier attention can easily overshadow any later tokens when LLMs calculate that those earlier ones are highly relevant. This all happens inside the embeddings and later the QKV matrices etc, completely independent from the actual tokens.

from my own very flawed perception, the very first thing (system prompt / first message) seems to carry a lot of 'weight', and the most recent prompt does as well, but the 'middle stuff' seems to carry 'less weight'. it would be very interesting to see this visualized or the 'mechanics'.

poloclub.github.io/transformer-... is a superb visual of the end-to-end GPT-2 architecture (modern architectures aren't _that_ different to this, so it's a good base).

seems too narrow, unless i didn;t click around enough. I understand how a single prompt or prediction works, but while using cursor/claude/chatbots I definitely feel that the first message must get re-injected or 'weighted' more somehow. Is it a 'harness' thing? do you experience this?

Interesting that the most consistently attended to tokens are newlines/periods. I think I've heard some folks argue that semantic information is getting stored for those tokens in the KV cache, not sure if that's true but there has to be some reason for that consistent focus.

Here's a version that lets me look at individual layers and heads (a layer just being the mean of all of its heads). Notice some heads really care about specific tokens that others don't.

Where can we try this tool?

The code is in a gist in this thread somewhere.