Post

We tested 25 approaches to MCP tool search, including decision models, LLMs, and embeddings. Our benchmark runs 150 search queries against 536 tools. We settled on weighted BM25 with a few normalization rules. Fast, local, no model or API calls.

that's what I landed on as well determinism good 😊

in theory, not really in practice

Well. Generally, you just set temperature to zero, give the same input on the same hardware and receive the same output.

you would think so, but even at a temperature of 0, major LLM providers do not guarantee identical results Anthropic actually has a disclaimer it's something to do with parallelism and scale, I don't fully understand it - some technicality makes big models somehow not fully deterministic 🤷‍♂️