Post

Finally had time to finish reading it, wonderful article as usual, the animations are so good!

Oh that is so good! Congrats on getting it out

Thank you 🙏❤️

The attention part was new to me. I’m trying to understand why and how they “project” now: how do they write a poem where the last word rhymes? They do and we know that, and my simplistic understanding is they somehow project forward. But how does that rhyme 🙌 with token by token?

I enjoyed the post, especially the interactive elements. Maybe an error or just a different preference, but it seems like the row labels in the attention weights matrix should all be shifted up one. For example it looks like ‘Mary’ has a 1.0 weight for ‘Mary’ rather than for ‘had’.

It’s not an error, it’s just that I very poorly explained it 😅 While the result of 100% is the contribution to the next token, the calculation is done by comparing the Q and K vectors of the token on itself. I didn’t really mention the term “self-attention” anywhere and that was a mistake.

That makes sense. You were covering a lot of extra ground already. Thanks again for the post.

Awesome post, incredibly well done as always. 💯 I know I’m in the minority with an iPhone Mini but I noticed a little issue on the smaller screen. The text in this demo wraps, causing height changes that make the rest of the article below jump up and down a bit.

Thank you ❤️ I did notice this while making it. Sadly that was the last visual I made and it was done the day before publishing, and I made the pragmatic decision to accept the imperfection. The smallest screen size I tend to target is the iPhone SE, is the Mini smaller? I might use that instead.

Hats off, dude. This is unbelievably well done!