Post

Smaller models forget system prompt instructions. What if we attach instructions to specific tools and deliver them the moment the model tries to use them? I call this guidance injection. samihonkonen.com/posts/guidan...

this is super cool! It definitely reminds me of deferred tools, but for a model without specific training for it. It also sounds like it can be implemented inference side, to prevent needing a round trip as well. I really like the ack as well

it should definitely be possible! deferred tools work like this inference side too to prevent round trips. As long as you know the tool->guidance mapping at api request time (chat completions, responses api, etc), you can just send both to inference and inference can manage the guidance flow

Mostly I just think this is cool and these patterns are useful at all layers of the model serving stack, and messing with the inference layer makes things fun :)

This is really interesting. Any tips on how to get started in experimenting with guidance injection at inference time?