Post

It feels crazy to me that there's no TypeScript-native, local-first tool for testing LLM-powered apps. So, I'm building one. It's based on Vitest, and it's called Evalite. Here's an early preview. www.aihero.dev/evalite-an-e...

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

"local-first Braintrust" sounds like a winner Braintrust's evals + logging has been a game-changer for me. It feels like the right mental model without having to adopt someone's crazy framework. Good luck!

100%. Not sure I'll have the heft to tackle observability and logging but hoping to make something compelling on the evals side.

the observability is so key I can see how my LLMs are actually performing against production inputs and then I can add a specific production input to a dataset with one click how might you match that workflow?

4. Good evals need traces, which let you view all sub-calls to LLM's inside your function and monitor approximate cost. This requires wrapping the test runner.

interesting, is this how it's done in the real world? I always assumed you would just write regular tests and seed the LLM to make it stable. LLMs are notoriously bad at scoring anything, aren't they? if your tasks and evals are both probabilistic, aren't the tests liable to randomly fail? 🤔

Yep, this is how it's done. LLM's don't respond well to regular pass/fail metrics. If you're asking what the capital of France is, there are dozens of ways the LLM could respond which 'pass' the test. "Paris" "The capital of France is Paris" "The city of lights."

still not sure what you mean by local first, are you running a small LLM locally for some tasks? (which one?)