Post

while I am anti-LLM, I did get lost training my own to test a hyper-specific focus. I also now see how hard the optimization problems are TLDR: this shit doesn't scale that is inherent to the discovery that bigger = better, and attention being quadratic complexity with few optimizations

16/3/26, 10:48 am0 reposts1 like

bread

@bread.codes

16 Mar

LLMs _will_ crash and burn economically all that will be left is the discovery of attention optimizations in ML, which is interesting, but also shows how dumb and inefficient these models really are and how attention tricked us into thinking the models know what we're saying

bread

@bread.codes

16 Mar

ML models should go back to being purpose built models that don't speak

Hurrdurr

@devhurrdurr.bsky.social

16 Mar

Not an expert, I’d like to hear your thoughts. Feels we hit the ceiling. Bigger models and context is nice, but one model for any question in any language doesn’t scale. Do you think they’re suitable for smaller tasks, eg. A to B translation, summarizing text. What do you mean “no speak”?

bread

@bread.codes

16 Mar

I'll try not to ramble. We have hit a ceiling, and may soon top-out modern architecture. The big discoveries that led to modern LLMs were: - Bigger = Better, which is antithetical to scaling - The "Attention is all you need" paper solving attention (so the LLM can "understand" things like pronouns)