Post

while I am anti-LLM, I did get lost training my own to test a hyper-specific focus. I also now see how hard the optimization problems are TLDR: this shit doesn't scale that is inherent to the discovery that bigger = better, and attention being quadratic complexity with few optimizations

LLMs _will_ crash and burn economically all that will be left is the discovery of attention optimizations in ML, which is interesting, but also shows how dumb and inefficient these models really are and how attention tricked us into thinking the models know what we're saying

ML models should go back to being purpose built models that don't speak

Not an expert, I’d like to hear your thoughts. Feels we hit the ceiling. Bigger models and context is nice, but one model for any question in any language doesn’t scale. Do you think they’re suitable for smaller tasks, eg. A to B translation, summarizing text. What do you mean “no speak”?

You came near the idea that we could make smaller "expert" models. Add a router layer and you have MoE (mixture of experts) which has made the models perform better. Every new model today uses MoE already.

MoE will get slightly (and I mean sub-20%) better with more high quality training data and more specific "expert" models, but they take a lot of VRAM to hold all the experts. Routing the experts from outside the model is hard without a huge inference time penalty, so they must stay as 1 huge model.

MoE is interesting and may be what tops-out last, but Attention is the other problem. The model needs to know who/what "he/her/it/they" refers to, and needs to know that "John" is the same "John" as you said earlier This issue is what dictates context size. 1M is crazy compared to where we started