Post

A coworker asked an agent to find issues in the project he works on and says that about 20% of the things it found were valid. Which is an awful lot of false positives but also it did turn up some real issues.

At the time, I was doing automated bug finding with human review. Human review found zero known false positives in the bugs I filed. Anthropic was using a better model and had more tokens to spend, which gave them the advantage on the two most difficult factors w.r.t. false positive rejection.