Post

contrastive-lm.notion.site Interesting write up. No paper yet unfortunately. If some of the claims hold true it would be a pretty incredible leap. Their SOTA DeepSWE scores come from generating rollouts using Opus/Fable and then using their model to select the "right" action.

They have the data and code out there, so can dig in to see how they did it I guess. Downside is that the SOTA scores would come from running 10x (or more) tokens on frontier models to get candidates.

Looks like Bo4 for the DeepSWE runs? huggingface.co/Contrastive-... In theory Fable/Opus Pass@4 should produce similar numbers right? I guess it depends if the model is choosing the final rollout results or being used mid stream to pick actions.

They did seem to train/fine tune on the DeepSWE tasks, with a set held out for the actual testing. Not sure what the fine tuning gets here, "style" for the outputs? Shouldn't the action model be general w.r.t. these sorts of inputs and not need fine tuning?

Seems like the fine tune is likely on the embedding space so that you get better contrast? Hence the usage of some of the DeepSWE tasks for fine tuning. I wonder what perf is like without fine tuning.

Also curious what it would look like to have this "in the loop" for a regular LLM, helping choose actions

For TBench 2.1, k=5 for the official run harness, I am curious if they used their model to pick the best of one of those, or if they picked the best of 4 or 5 runs of k=5, essentially getting k=20 (or k=25).