Post

Sam Rose
@samwho.dev

New post! ✨ Prompt caching ✨ My first big project post for @ngrok.com. 5 weeks, 12217 lines of code, 195 commits. I poured a lot into this one, and learned a lot in the process. I really hope you enjoy it ❤️

ngrok@ngrok.com

Yesterday we launched ngrok.ai into early access. Today we're bringing you a deep dive into LLM internals with beautiful visuals crafted by our very own @samwho.dev. Discover exactly what gets cached to offer you 10x cheaper input tokens. ngrok.com/blog/prompt-...

klievhelm.bsky.social
@klievhelm.bsky.social

I enjoyed the post, especially the interactive elements. Maybe an error or just a different preference, but it seems like the row labels in the attention weights matrix should all be shifted up one. For example it looks like ‘Mary’ has a 1.0 weight for ‘Mary’ rather than for ‘had’.

Sam Rose
@samwho.dev

It’s not an error, it’s just that I very poorly explained it 😅 While the result of 100% is the contribution to the next token, the calculation is done by comparing the Q and K vectors of the token on itself. I didn’t really mention the term “self-attention” anywhere and that was a mistake.

0 reposts0 likes

Sam Rose
@samwho.dev

You’re very welcome. It really helps me improve when people give this sort of feedback, I greatly appreciate it. 🙏