Post

Assessing LLM output quality requires managing their vast input/output spaces and acute sensitivity to minor prompt changes, but many teams are still running on largely vibe-based evals, incurring considerable risk and preventing systematic product improvement. We can do better! #databs #mlsky

I recently completed @hamel.bsky.social and @sh-reya.bsky.social's AI Evals For Engineers & PMs, which presented practical frameworks for identifying appropriate dimensions of quality, creating automatic validators to measure at scale, and effective feedback loops that enable continual improvement

The course validated my intuitions that these foundations apply just as much to generative AI applications and has given me a road-tested pragmatic framework for applying them to the weird world of LLMs and AI engineering.