Post

What do LLMs see? I wrote a lil' tool that extracts the attention matrices out of open models and creates this typing visual, with each token's opacity changing according to its average attention score as the prompt progresses. Dimmer words are considered less important to the model.

This was done with a relatively small model, LFM2.5 1.2B. In the process of downloading the biggest model my lil laptop can run to see how it looks.

Also, worth noting: this is not super representative of what the LLM actually thinks of each token. It's a global average from every attention layer and every head, which "blurs" the detailed information. It also doesn't consider the projections that happen after attention. It's just a bit of fun.

I wrote it because my boss nerd sniped me by saying "recent tokens in context are always much more important than old ones." I wanted to know if that was actually true. So far I think it _might_ be, but more testing required.

Here's a version that lets me look at individual layers and heads (a layer just being the mean of all of its heads). Notice some heads really care about specific tokens that others don't.

Where can we try this tool?

The code is in a gist in this thread somewhere.