Post

Rasmus Aagaard
@rasgaard.com

Any examples of orgs creating an internal SWE-bench? Curious about having ways to measure when local models reach a “good enough”-stage to be viable of x% of relevant coding tasks and this seems to be a promising way to communicate capabilities internally

2 reposts6 likes

Vladimir Salnikov
@v4ldelund.bsky.social

I am in the process of creating one (although quite lazily and in robotics specific tasks) Def not enough information out there, maybe after internship finished I will make one for myself and will share a blog

John Brown’s Body
@berndtpetersen.bsky.social

Just take existing benchmarks, pick the model in the past where you first relied on closed models, and match the benchmarks they hit. For most of us, we passed the point a while back where the local models can beat the performance of what we relied on before/first.