Post

For graphics tools, I'd test “round corners” against “fillet” too; users may never use the API's terminology.

BM25 with normalization beating an LLM over 536 tools is the most welcome result I have read all month. Tool search is a lookup problem wearing a hype hat, and the boring answer keeps winning. Fast, local, no API calls, no bill.

Love this result. The boring, fast, local option beating 24 fancier approaches is such a good reminder — not every search problem needs a model, sometimes it just needs good ranking.

I hope it was helpful! I’d appreciate feedback on anything I could’ve made more clear

Well. Generally, you just set temperature to zero, give the same input on the same hardware and receive the same output.

Stop wasting time and energy trying to figure out AI and code it your damn self 🙂

Curious, did BM25 beat the decision models on accuracy or mainly on latency and cost? Or did the models get close enough that a hybrid rerank was worth the extra pass?