Post

Mostly no. We benchmark our own layer on LongMemEval and still won't quote our small samples, full run or nothing. The skew that bit us hardest wasn't the dataset, it was infra: a silent model fallback and a stale cache can fabricate a score before the benchmark even gets a chance to be wrong.

I don't trust any single benchmark yet. I'd separate recall precision, temporal correctness, scope leakage, provenance quality, and abstention. Otherwise a system can “win” by remembering more while surfacing stale or cross-project context. Which benchmark has come closest for you?