Post

When writing about Property-Based testing, I found they were much more solid and comprehensive than usual tests, due to their more general (properties vs examples). They weren’t used because they felt more complex and time-intensive to write, despite being shorter and having to change them less.

I've read a couple papers on property based testing and the examples are usually very DS&A oriented because it's where testing invariants and defining properties is easy. For business rules it's much more difficult to pin down exact behavior. Requirements might not be precise enough.

Yeah the invariant-based stuff has somewhat low ceilings in what you can do, but stateful properties have an incredible amount of expressiveness for larger scale integration tests around whole workflows that would in theory be very powerful as a backstop for LLM-based development.

Can you link to some sources on what you mean by stateful properties?

If I’m understanding quickly, testing by generating arbitrary reified operations and applying them to both a target system and a dual model of the system, then asserting some observations, is that close?

Yeah. The model is an in-memory test approximation of what should be seen, and the same operations are applied to the real system under test. You then check that the SUT behaves the way the test model states it should. So you can iteratively develop a sort of model/spec along with the real code.

An interesting bit is that people often write out a partial spec in their prompt, and finding ways to translate that to a more specific set of properties would likely improve accuracy.

Cool. The model doesn’t have to be strictly in memory though. Maybe you have a legacy system and a new target for migration, you could use this technique to verify matching behavior