Post

Anyone have a good guide / reference on designing actually good skill evals?

0 reposts0 likes

Not a guide, but what helped us: test the skill's output, not the transcript. Write one small case for each failure you have actually seen. Assert hard facts first (right tool, valid args, no out-of-scope writes). Grade quality separately. Run them on every skill change.