Post

Models like Talkie-1930 sound like voices from the past. If they could reliably speak from specified historical vantage points, researchers might also use them to simulate the past. But how reliable are they? Today we release a benchmark answering that question for English contexts 1831-1930.

Measuring factual knowledge 𝘢𝘣𝘰𝘶𝘵 the past isn't hard: multiple choice questions work. Measuring a model's ability to behave 𝘭𝘪𝘬𝘦 someone in the past is much harder. What a person would say depends on where they stood, so we pair our 866 questions with "metadata frames" that specify a context.

Reasoning models are better at recognizing plausible answers than at producing them, so we don't give this as a multiple-choice test. With open-weight models, we can directly compare the likelihoods of good and bad answers. Models trained on historical text excel on this version of the test.

We can also just ask models to generate answers. Free-text generation is challenging to judge; the tldr is that we do it w/ Elo-style comparisons. Scores from this version of the benchmark below. Talkie-1930-13B-it outperforms some much larger models on style, but Astra performs better overall.

Does this mean history is solved — just use Astra? Definitely not. Our "constrained generation" score comes closest to measuring what researchers might ask models to do (speak from a specified historical position), and human validation suggests a score of 71.4 is still perceptible as pastiche.

Also, the bar for research would be higher: models would need to avoid even errors imperceptible to a modern eye. We'll keep making our benchmark harder. But the 1.0 version already shows where models have improved, and where they need more work. Blog: historical-ai-lab.github.io/Chronologic-...

What do you mean by “researchers might ask models to do (speak from a specified historical position)”? And what research would one with these models?

We discuss that a bit in the first paragraph of the blog post above, and a bit more on the first page and last page (p.8, pre-references) of the paper. But they’re examples touched lightly, and one of the points we make on p. 1 is that we’d need a benchmark even if LLMs were never used for rsch. +