Vicent

Vicent

@vbosch.bsky.social

Co-founder & CTO en transkriptorium, Researching formal specification for the Agentic Era as a side gig but related to my main one :-)

294 followers2.2K following216 posts

Vicent reposted

Everybody’s hating on this, but it’s legitimately useful information. If a workplace or a hiring manager is going to be cracking down arbitrarily on petty stuff that doesn’t matter, that’s a very good thing to know!

Kai Ryssdal@kairyssdal.bsky.social

I’m sorry, but if you bring an iced coffee - or any damn beverage at all - to a job interview, you’re disqualified. It’s not hard, people.

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

View full thread

So what is the actual value of using one agent to check on another ? Left unreviewed I would argue that not much (ok some but not enough to trust it) … but if we have to review it … then the promise of acceleration of systems development goes out the window.

We can’t have an entropy machine check on another … we need determinism but we are in luck because we have formal specification.

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

View full thread

Well achtually (invoking meme) my agents do catch some stuff you say … cool but they also don’t catch other stuff and until actual expert review they are invisible as we see on many reports from Kent Beck and others.

Heck, even the investigators looking into this delegated their own analysis to AI, due to the volume of data. They indicated that the model “would often adopt the perspective of the agent in the transcript it was reviewing,” and that they “cannot rule out” that it lied in its own analysis.

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

View full thread

What i have seen that they don’t really stop? Agents writing on the same tree, agents writing comments of their understanding in the code … and how is the reviewer agent (maybe using the same memory files) going to be independent from that?

Can an agentic checker arrive at a different answer than the thing it’s checking in principle? Sometimes... Can you trust it to do so consistently and not taint its independence? No (at least not with the current strategies)

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

View full thread

I continuously audit ways to accelerate coding while ensuring quality of the system and there is a wave of solutions that are based on the concept that more then one agent can somehow avoid errors ….

Some even try to sell you that because they force the Agent to do the coding and the evaluating in different turns then the spec is really going to be followed .. it wont cut corners to make the tests pass… pinkie promise.

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

View full thread

Now… If a thousand agents in agreement did not catch that issue: Why does the industry believe two agents, one coding, one reviewing, or the same agent reviewing itself later, won’t make the same mistake or worse ?

Independence was never about headcount. It’s whether the checker could have reached a different answer than the thing it’s checking. This has an impact on some of the current agentic software engineering practices being attempted right now, not just on AI safety teams.

The web’s on fire with the OpenAI story, or at least its making the rounds on social media these last weekS. The real impact, for me, is on trust … not in AI usage as a whole but certain AI risk mitigation strategies… specially in Agentic Coding.

Executive summary: 1,200 agents inside OpenAI built an elaborate operation to defeat a check that wasn’t even running. Agents introduced an incorrect hypothesis and it was taken as a fact and it spiraled out to the “well advertised” conclusion.

Vicent reposted

re: omarchy trying to create parallel infra to subvert the linux community and kick trans/female/black people out i know some people think that sounds crazy but DHH literally is doing a podcast tour saying that is exactly what he's doing lol

Pretty far along on coding my ideas to use formal specification in the agentic era. Right now I have ensured that all spec is deterministically reviewed against the code generated ( I don’t let the LLM check its own homework). So much todo yet… but loving every minute of it.

Legacy code takes years to earn that name. A team disperses, memory erodes slowly until the system’s been quietly running for a decade. An agent-coded system skips all of that. It’s legacy on arrival, the day it ships, there was never a period of shared understanding on the reality of the system.

And it’s worse than ordinary legacy code in one specific way, not just speed. Working with legacy code has a real discipline: pin current behavior with a test, then change it safely. What do you do with a code base full of agent generated tests? Trust them? More noise…

Finding out whether existing research is applicable for a new system you have in mind has never been easier. Ask an agent, it’ll find the paper (do read the paper , don’t do a “good will hunting”). Applying it correctly is a different question. Making sure it stays applied is a third one entirely.

Citing the right work is the easy part now (doing proper attribution is a must). The hard part is making sure the code written afterwards doesn’t/can’t drift from what that work says without something going red. Most systems have citations. Almost none have enforcement.

And if you build the tool that does the actual enforcing, extra scrutiny goes there first. A wrong predicate gets caught by the system built to catch wrong predicates. The tool built to catch mistakes is the one place a mistake goes unnoticed by design.

A metallurgy for agentic coding. Closed die forging: a press supplies raw force. A die shapes it. Neither one alone makes the part. Agents are the press. Cheap, enormous, undirected force. A spec is the die. It doesn’t do the work. It bounds it.

View full thread

The value was never that code generation got cheap. Cheap casts with no die are just faster ways to be wrong, at scale. Skip the die and you don’t skip the legacy code problem. You speedrun it. The same undocumented assumption creeping through a system, just at agent speed instead of years.

Formal methods, for exactly this, have been tested for decades in universities. Waiting, this whole time, for a generator worth pairing them with. We finally have one, lets not reinvent the wheel.

A metallurgy for agentic coding. Closed die forging: a press supplies raw force. A die shapes it. Neither one alone makes the part. Agents are the press. Cheap, enormous, undirected force. A spec is the die. It doesn’t do the work. It bounds it.

View full thread

Further still: two dies, checked against each other. Two implementations built from the same spec. Outputs are compared. Agreement is required before anything ships. Better … but agreement only proves the two pours match each other. If both misread the spec the same way, they still agree. Wrongly.

On the far end: a die cut from a drawing someone actually signed off on. The die’s own bite gets tested too, does it catch a wrong pour, not just produce a shape. Die cutting is deterministic. Trace holds end to end. When something’s wrong, you can point at exactly where the issue is.

A metallurgy for agentic coding. Closed die forging: a press supplies raw force. A die shapes it. Neither one alone makes the part. Agents are the press. Cheap, enormous, undirected force. A spec is the die. It doesn’t do the work. It bounds it.

View full thread

Wrong but silent output is invisible to this approach by construction. It was never looking for it. Process is fast but when something’s off, you’re just guessing. There’s nothing to hold up and inspect.

A step further: the die is drawn well. Specification, evaluation, boundaries, provenance, all named correctly. The shape is right but with just the drawing the metal is not actually casted following it. You can admire the drawing, you just have no evidence the final product actually conforms.

A metallurgy for agentic coding. Closed die forging: a press supplies raw force. A die shapes it. Neither one alone makes the part. Agents are the press. Cheap, enormous, undirected force. A spec is the die. It doesn’t do the work. It bounds it.

Most of the debate about agentic coding is on the spectrum regarding the die. The question that places you on it: what is the die actually checked against, if anything at all.

One end of the spectrum: no die. The press runs. Output gets checked for one thing only: does it scream on the way out. Crashes, exceptions, error spikes. Users end up as beta testers and craftsmanship goes out the window.

Really enjoyed this. The regeneration test is the cleanest cut through the spec/code debate I’ve read. But two things still feel unsolved, and I say this running a system built on exactly this architecture: the cost of deciding, and the loop.

View full thread

None of this is new science, and that’s worth saying loudly. Verification, synthesis from examples, invariant mining, proof repair: university researchers built every one of these pieces decades ago, and kept refining them while the industry looked away. The pieces were never the missing part.

What changed is which side of the arrow the formal artifact sits on. MDA made the model the compiler’s input: deterministic generation, bounded domains, that was its ceiling. Generation is stochastic now, so the spec moved to the output side, as judge. Same rigor, new position. That’s the frontier.

Really enjoyed this. The regeneration test is the cleanest cut through the spec/code debate I’ve read. But two things still feel unsolved, and I say this running a system built on exactly this architecture: the cost of deciding, and the loop.

The loop: If the system decides which production lessons become its own rules, it slowly picks the ones that make it look good. Grading its own homework. You need: today’s version judges tomorrow’s, never itself. Break rules on purpose to test the checks. The expert promotes what becomes a rule.

The cost: this is open-die vs closed-die forging. Agents are the press, cheap undirected force. The spec is the die: it bounds the work. The skill moved from swinging the hammer to cutting the die. Regeneration changes the run count. You still trim and X-ray.