Post

It feels crazy to me that there's no TypeScript-native, local-first tool for testing LLM-powered apps. So, I'm building one. It's based on Vitest, and it's called Evalite. Here's an early preview. www.aihero.dev/evalite-an-e...

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

3. I'm using local-first to compare against other services like Braintrust, which require an API key to do anything. Evalite will produce a local UI, no API key required.

"local-first Braintrust" sounds like a winner Braintrust's evals + logging has been a game-changer for me. It feels like the right mental model without having to adopt someone's crazy framework. Good luck!

100%. Not sure I'll have the heft to tackle observability and logging but hoping to make something compelling on the evals side.

the observability is so key I can see how my LLMs are actually performing against production inputs and then I can add a specific production input to a dataset with one click how might you match that workflow?