Post

New preprint: "Unstructured Evidence Attribution for Long Context Query Focused Summarization" We propose *unstructured* evidence attribution for long context summarization and a synthetic dataset called SUnsET which can help models perform this task. Paper link: arxiv.org/abs/2502.14409 Thread ⬇️

LLMs are really good at summarizing long contexts. Recent work has looked at citing specific evidence within a summary. But these works use fixed granularity evidence (e.g. sentences or paragraphs)! We propose to extract and cite free text spans from the context to improve flexibility.

We want to be able to adapt LLMs to this task. The ideal dataset would be extremely difficult to curate by hand: long documents paired with questions, summaries, and free text evidence from the context. To get around this, we create ☀️SUnsET☀️: The Summaries with Unstructured Evidence Text Dataset

SUnsET is created using a novel inductive synthetic data generation pipeline. Each stage has carefully engineered prompts designed to maximize diversity and accuracy. The dataset consists of documents broken into multiple sections, with multiple queries and summaries which cite specific text spans.

We use SUnsET to adapt LLMs to the unstructured evidence attribution tasks using two approaches: standard fine-tuning and fine-tuning on shuffled documents, in order to overcome the lost-in-the-middle problem for evidence attibution.

We have the following main findings across 5 models and 4 disparate test datasets: 1) All base models struggle both to extract evidence text and to use it correctly. 2) Fine-tuning on SUnsET improves model ability to extract and correctly use evidence across the board ⬇️

3) Evidence tends to be lost-in-the-middle for all base models 4) Shuffling the document sections helps mitigate evidence being lost-in-the-middle 5) Learning to cite with SUnsET also improves the quality of the final summaries