Post

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

• Helpfulness: Usefulness (directly addresses the question with enough detail and explanation) and completeness (does not omit important details); an answer can be faithful but not helpful if too brief or doesn't answer the question

• Evaluate separately: Faithfulness = binary label -> LLM-evaluator; Helpfulness = pairwise comparisons -> reward model

2. How to build robust evals • Use LLMs to generate questions from the text • Evals should evaluate positional robustness (i.e., have questions at the beginning, middle, and end of text)

I like this fusion of “usefulness” (which is a better way of describing what “relevance” tries to get at imo) and “completeness.” They both go together but it also sidesteps some challenges with “completeness” as a metric in certain domains (preexisting use of it creates baggage)

thank for for the feedback! it’s an initial attempt at defining helpfulness as a combination of both and can be improved. all ideas appreciated