Post

The more I work with LLMs and agents, the more I'm gobsmacked by the n^2 nature of how conversations are handled. For every turn of the conversation, you send the entire previous conversation. This means the costs compound the longer the conversation goes on.

That sounds like a vast underestimate. When we did semantic search a couple years ago, each embedding was ~3kB, and I imagine the preprocessing of a prompt consists of a single embedding _per token in context_. Embeddings these days may have a different size, but probably not vastly smaller

Of course once it makes its way to the LLM things blow up as each input token fans out to encode embeddings. Tokens are presented as very large one-hot vectors to the model but then translated to floating point lower dim embeddings. Higher info density, lower dimensions.

1 token -> 200k one-hot (~25KB) 200k one-hot -> 4096 floating point (~65K @ 16bit FP) (Just doing the math for my own education as I'm brushing up on this stuff 😊)