Post

the fundamental issue with AI has ALWAYS BEEN that it CAN'T do precisely what you intend because prompts are lossy compression you can't control the decompression of. that is what the technology **IS** now they are rebranding that as "lack of alignment" as if it wasn't present from their inception

Something I find frustrating is that I think LLMs are incredibly useful. However, the problem isn't necessarily their capabilities, it's that people grossly misunderstand and don't attempt to actually understand the capabilities.

I did end up paying $100/mo for Claude and I was having a blast. But that pretty quickly ended when I realized that it is only as good as I am. I still need to put in the effort to do the engineering behind whatever I create. I need to do the fact checking. I need to understand the commands.

I think they are useful as semantic search engines, especially if you're trying to navigate your own data. But I think we need regulations for standardization of performance expectations and holding companies accountable for grossly over promising the capabilities of these things.

And given all of this, I think one way to achieve that is to do the enforcement of copyright that we've been wanting this whole time. Forcing companies to have to respect copyright and suing them into the ground would force these models to slow down and our expectations to be curbed.

but it can't do that either. a huge part of the actual problem is selling a product on the idea that it understands things when it is functionally incapable of that "how many R's in strawberry" and all that

That's the thing though, is that in my experience it statistically is pretty solid at getting certain types of information so it is useful in some ways. It's useful as a tool to semantically try to find very specific information and then take its results and then research manually.

It allows me to kind of skip steps in searching where I typically would need to filter through dozens of stack overflow hosts that only maybe get kind of close to what I'm looking for. But the core problem is that most people don't know how to use it like that. They just trust its output.

One other factor: it really depends on what model you're using. Gemini flash, which is the same one used in the Google AI summary, is garbage compared to Claude Opus 5. It's actually to the point that I'd argue AI summaries in search engines should be illegal for misleading the public.