Post

Symon Baikov
@symonbaikov.bsky.social

Benchmarks like this are useful when they separate bug-fixing from full task ownership. Agents can look impressive on contained edits and still struggle when the work requires design recovery, missing context, and long-horizon judgment.

Florian Schepp
@scheppening.com

It's always somewhat relative and differs to tasks you'll encounter as part of your individual teams but this is exactly where DeepSWE is more realistic including real sometimes messy open source GitHub issues where the missing context and design recovery are part of solving the particular issue.

Symon Baikov
@symonbaikov.bsky.social

That is the right direction for benchmarks. Real engineering tasks are messy because the hard part is often reconstructing intent from partial docs, stale issues, and repo conventions — not just patching the obvious failing line.