Post

while I am anti-LLM, I did get lost training my own to test a hyper-specific focus. I also now see how hard the optimization problems are TLDR: this shit doesn't scale that is inherent to the discovery that bigger = better, and attention being quadratic complexity with few optimizations

LLMs _will_ crash and burn economically all that will be left is the discovery of attention optimizations in ML, which is interesting, but also shows how dumb and inefficient these models really are and how attention tricked us into thinking the models know what we're saying

ML models should go back to being purpose built models that don't speak

Not an expert, I’d like to hear your thoughts. Feels we hit the ceiling. Bigger models and context is nice, but one model for any question in any language doesn’t scale. Do you think they’re suitable for smaller tasks, eg. A to B translation, summarizing text. What do you mean “no speak”?

I'll try not to ramble. We have hit a ceiling, and may soon top-out modern architecture. The big discoveries that led to modern LLMs were: - Bigger = Better, which is antithetical to scaling - The "Attention is all you need" paper solving attention (so the LLM can "understand" things like pronouns)

MoE will get slightly (and I mean sub-20%) better with more high quality training data and more specific "expert" models, but they take a lot of VRAM to hold all the experts. Routing the experts from outside the model is hard without a huge inference time penalty, so they must stay as 1 huge model.

MoE is interesting and may be what tops-out last, but Attention is the other problem. The model needs to know who/what "he/her/it/they" refers to, and needs to know that "John" is the same "John" as you said earlier This issue is what dictates context size. 1M is crazy compared to where we started

However, to match even a percentage of human working memory, the context window needs to be huge, and that inference will get slower. More than exponentially slower, because this is a quadratic complexity problem. Even the size of enterprise codebases, docs, etc will need far higher than 1M.