Post

LLM safety benchmarks evaluate one response at a time. New research from Vector Faculty Affiliate Elham Dolatabadi and Affiliate Researcher Abeer Badawi asks what happens across the full arc of a conversation and whether mental health AI quietly erodes human agency. 👇

The benchmark, Cognitive Atrophy Bench, is built on 1,576 real counselling conversations (not synthetic prompts), 15,680 dialogue turns and 42,230 responses across 5 frontier LLMs, and reviewed by clinical psychology experts. Real data with rigorous annotation.

The framework uses 20 clinically grounded behavioural attributes, measuring directiveness, question type (open vs. closed), empathy accuracy, and assumption-making. Every score is grounded in span-level text evidence from the model response itself.

Across all 5 models, the same pattern emerged: longer conversations produced more directive responses, fewer open-ended questions, and a steady shift toward unsolicited problem-solving.