Post

Philip Nuzhnyi
@callmephilip.com

Each question is framed as a _numbered_ multiple choice question. E.g. Options like "blue, red, yellow" become 0, 1, 2. The model will answer with the number.

Philip Nuzhnyi
@callmephilip.com

Figure out model's internal encoding representation for the IDs your system supports (up to 255 choices in this case). This extra PREFIX business seems to be dealing with possible different ways to encode the same value (e.g. "2") depending on whether it appears in "2" or "choice_index: 2"

Philip Nuzhnyi
@callmephilip.com

Massage your prompt so it looks like "user says / assistant says" with our prefix at the very end so that the model is set up to spit out a single value for one of the choices.

Philip Nuzhnyi
@callmephilip.com

! Do 1 round through the model (as opposed to autoregressive loop where you feed results from the previous batch into the next iteration and keep assembling final result). Only grab logits for the last chunk (what comes after "choice_index:" in our case). Important: send multiple questions in 1 go

0 reposts0 likes

Philip Nuzhnyi
@callmephilip.com

Get rid of any results that are not our supported choice indices. Turn logits (random looking numbers like 3, 89, -386) into a list of probabilities that add up to 1. Get your answers with probabilities back.