Post

Smaller models forget system prompt instructions. What if we attach instructions to specific tools and deliver them the moment the model tries to use them? I call this guidance injection. samihonkonen.com/posts/guidan...

samihonkonen.comGuidance injection: reliable instructions for local LLMsSystem prompts don’t work reliably for smaller models. Guidance injection delivers instructions at the exact moment they’re needed instead.

Alec Solder

@alec-s.bsky.social

24 June

this is super cool! It definitely reminds me of deferred tools, but for a model without specific training for it. It also sounds like it can be implemented inference side, to prevent needing a round trip as well. I really like the ack as well

Alec Solder

@alec-s.bsky.social

24 June

I think that using this to keep the system prompt small could lead to the model ignoring the guidance in tool call output due to the instruction hierarchy and anti-jailbreak. Deferred tools schemas are appended as system/developer messages iirc

Sami Honkonen

@samihonkonen.com

Interesting! I hadn't thought about instruction hierarchy and so far haven't seen this come up. Maybe it's because I have generic tool use instructions in the system prompt and the injected guidance makes it more specific. Maybe the mentions in the system prompt lend credence to the tool outputs?

24/6/26, 10:09 am0 reposts1 like

No replies yet.