Post

It feels crazy to me that there's no TypeScript-native, local-first tool for testing LLM-powered apps. So, I'm building one. It's based on Vitest, and it's called Evalite. Here's an early preview. www.aihero.dev/evalite-an-e...

I don't understand what this is. 😅 it looks like it's just mapping your input/expected over a function? why do you need a custom test runner for this? and how is it local first? most useful models are too large to run locally. some sort of API response cache/recorder is probably what I need? 🤔

Some background on evals, full explanation below: www.aihero.dev/what-are-evals

1. First, evals return a score, not a pass/fail metric. So you need a custom test runner to see the scores. 2. Good evals also need a UI attached so you can view the results, since you often need to verify the LLM app's output with human eyes.

3. I'm using local-first to compare against other services like Braintrust, which require an API key to do anything. Evalite will produce a local UI, no API key required.

4. Good evals need traces, which let you view all sub-calls to LLM's inside your function and monitor approximate cost. This requires wrapping the test runner.

interesting, is this how it's done in the real world? I always assumed you would just write regular tests and seed the LLM to make it stable. LLMs are notoriously bad at scoring anything, aren't they? if your tasks and evals are both probabilistic, aren't the tests liable to randomly fail? 🤔

Yep, this is how it's done. LLM's don't respond well to regular pass/fail metrics. If you're asking what the capital of France is, there are dozens of ways the LLM could respond which 'pass' the test. "Paris" "The capital of France is Paris" "The city of lights."