Post

spooky Deirdre Connolly¹ ² at a distance

"LLM-generated patches did not resolve the vulnerability, added a new vulnerability, or both, an average 53.9% of the time." 👁️👄👁️ 1password.com/blog/why-ai-...

6/8/26, 8:30 pm120 reposts174 likes

Glyph

@glyph.im

6 Aug

@durumcrustulum.com makes me feel foolish for even conceding a little bit that LLMs are useful tools for analysis

spooky Deirdre Connolly¹ ² at a distance

@durumcrustulum.com

6 Aug

they are useful, but must be used with care

Glyph

@glyph.im

6 Aug

@durumcrustulum.com I am not confident enough to say “they’re not” but I am also really not confident enough at this point that they pass the futzing fraction test either, much like every other use case

Lovasoa

@lovasoa.mastodon.social.ap.brid.gy

6 Aug

@durumcrustulum.com Looking at the article it sounds like the sample size is tiny: 6 handpicked vulnerabilities. A better title would be: out of thousands of CVEs, we found six that frontier LLMs could not reliably patch.

eufalconimorph.bsky.social

@eufalconimorph.bsky.social

7 Aug

True, but they needed vulnerabilities new enough to not be in the training datasets, and nontrivial to fix, while also being in open-source projects which were in the training datasets. That certainly limited the sample size.

T045T

@t045t.zmb.cm

11 Aug

sounds like they *also* read through all the patches and evaluated the code quality, so that also limits the scope. anecdotally, a slightly above 50% chance of producing "fix-like artifacts" (love that term) sounds about right, though.

Rubyloon Foxo

@foxyloon.blimps.xyz

14 Aug

@durumcrustulum.com Statistically, that suggests the output it generates is completely random. Go figure...