Post

hailey
@hailey.at

i think the assumption that LLMs are going to make software more stable and secure are basically for the foreseeable future outright false. LLMs are not producing more secure and stable code unless significant attention is paid to what they ship, which it generally is not. and attacking is ~free.

conputer dipshit@davidcrespo.bsky.social

would be interested to see more debate on this from security types and even some formal modeling. my hunch is that you can lock down individual systems quite well and the ways they interact in such a way that the probability of end to end exploits goes tolerably close to zero x.com/CJHandmer/st...

hailey
@hailey.at

they are great at finding and fixing existing bugs. they are not great at not introducing new bugs, and its kind of mind boggling that such a wide range of people seem to think otherwise.

5 reposts92 likes

nomantis
@nomant.is

And this is not an LLM problem btw, it's a software engineering problem. It existed before too, it's just exacerbates it.

nomantis
@nomant.is

But for realsies, I see two huge problems: People who blindly trust the output, and just prompt->publish. That's just irresponsible in any online setting, where the service has any attack surface. Which is basically all software that is not inside an airgapped system.

nomantis
@nomant.is

Second is more nefarious: making changes without enough context of the problemspace. Even if you're the most vigilant developer, if the change made covers a large enough area it's exponentially more difficult to rule out a bug. It's basically an endurance game at some point.

nomantis
@nomant.is

Are you going to look over every change site, determine the effects of the change (primary, secondary, maybe even tertiary effects)? Or will you look closely at the first 5-10 and lose focus on the rest? And at this point we're not even talking about time pressure and company expectations.

philpax
@philpax.me

new bugs are existing bugs that haven't aged yet. with that being said, yes, the people doing prompt-to-publish are unlikely to be pulling out all of the stops

Laurens
@laurenshof.online

Yeah. Plus we’re entering an entirely new class of software where there is deep ai integration into the system itself and we have not even begon to explore what all sorts of new vulnerabilities and risks and bugs this introduces

Stefano Zanero
@raistolo.bsky.social

Mostly agree. I would say they are much better at finding and exploiting bugs, because vulns are testable counterexamples. That’s a setting where you can let the generative AI ruminate and if the result happens to work it doesn’t really matter why. Fixing is a whole different story.

Cyborg Tribe
@cybrogtribe.bsky.social

In my experience this probably isn’t true in practice. You just stick the bug finding AI into CI, and so while yes, a single shot coding agent doesn’t produce secure code, people can still ship generally secure code. But I do think we won’t know either way with any big certainty for years.