Post

Josh Black
@josh.black

It's been wild catching up on LLM stuff after being out on parental leave for a bit 😵‍💫 One of the biggest things that came up when working on Primer for me was trying to find a way to answer the question "does this make things better, or not?" This lead to a fun project: github.com/primer/agent...

0 reposts3 likes

Josh Black
@josh.black

So many LLM resources were flying around, from MCPs to skills to instructions to sub-agents 😅 I really wanted to find a way to create experiments to compare these different ways of doing things against scenarios that I cared about (and could deterministically test against)

Josh Black
@josh.black

Under the hood, all this really does is bring in the different scenarios into a fresh Docker container and run the prompt using Copilot CLI. But it was cool to start seeing how different performed across not only the scenarios but against different models too

Josh Black
@josh.black

It will be interesting how useful this will be looking forward 👀 It feels so helpful for making sure tools we ship actually do _something_ in terms of improving outcomes. I also hope it will not only let us make helpful additions over time, but also removals as models change over time

Josh Black
@josh.black

Super curious what other folks are doing in this space 👀 If you have a project or technique that you've found helpful would love to hear about it! I would imagine we're all on some part of this same journey as we figure this out for our own design systems