Post

What do LLMs see? I wrote a lil' tool that extracts the attention matrices out of open models and creates this typing visual, with each token's opacity changing according to its average attention score as the prompt progresses. Dimmer words are considered less important to the model.

This was done with a relatively small model, LFM2.5 1.2B. In the process of downloading the biggest model my lil laptop can run to see how it looks.

Also, worth noting: this is not super representative of what the LLM actually thinks of each token. It's a global average from every attention layer and every head, which "blurs" the detailed information. It also doesn't consider the projections that happen after attention. It's just a bit of fun.

I wrote it because my boss nerd sniped me by saying "recent tokens in context are always much more important than old ones." I wanted to know if that was actually true. So far I think it _might_ be, but more testing required.

> recent tokens in context are always much more important than old ones The key insight to overcome this fallacy is to stop thinking in tokens and accept that LLM "think" in embeddings, which contain multiple tokens plus their attention "score".

Is it a fallacy?

A combination of tokens inside those embeddings plus earlier attention can easily overshadow any later tokens when LLMs calculate that those earlier ones are highly relevant. This all happens inside the embeddings and later the QKV matrices etc, completely independent from the actual tokens.

Can you show me this happening empirically?

The process can get partly visualized, I think, see below.

I've seen that video, it's not really about attention though right? It's how models learn during training. Can you point me to a specific timestamp in the video that supports what you originally said in the thread?

The first step maps tokens to embeddings 1:1, and then from there they get passed into an attention layer. It's very reasonable to think of those embeddings, at least in early layers, as tokens. I'd have to dig up where I read it but early layers often do "token merging" so it gets fuzzy further in.

It is correct that the first step is a 1:1 mapping. But once the first attention step is passed, they lose their mapping, which is whole point. The weights control relevance from that on, not the position, so later tokens can and will get completely ignored (or "downvoted"), if necessary.

Those weights can of course amplify later tokens, if e.g. a a relevant instruction comes at the end of a prompt and not at the start. But that is an artifact from training. Nothing else in LLM/ the attention mechanism prefers earlier or later tokens outright.