Post

Jeremy Smith
@jeremysmith.co

I'm reviewing all agent code for client projects and having good success lately. Very few bugs or defects. If we're reworking things, it's because I didn't spec clearly enough or changed my mind after seeing the code. But, I've had one area of repeated failure in recent weeks…

Jeremy Smith
@jeremysmith.co

And that's on a project where I'm slowly migrating complex authorization rules from CanCanCan to ActionPolicy. During review, I've extracted relevant rules from both sides and walked the authz logic myself, and in this way I've caught multiple mistakes in the translation of rules.

0 reposts2 likes

Jeremy Smith
@jeremysmith.co

Now I'm stopping to build a better migration process (including a set of comprehensive bridge tests). Just thought it was a good data point where code has the appearance of correctness (easy to just LGTM), test coverage is higher, CI is green, and you're still in a failure mode.

Jeremy NT
@jeremynt.bsky.social

This is a good cautionary tale because it's something so critical... did you use an agent to review the PR first and if so did it also miss the issues?

Jeremy Smith
@jeremysmith.co

Yes, my typical process has been to have another agent review the PR. And for this, I've used 2 different reviewing models. This project has been intermittent over the past few weeks. But from what I can remember, reviewing agents might have caught a couple issues, but I've manually caught 3-5 more.

Jeremy NT
@jeremynt.bsky.social

At least you got enough signal from the agent review to indicate there were dragons in there! If I get a clean agent review I'm likely to perform a less thorough manual review than I might have otherwise...