Post

Henry🔸
@henryjosephson.com

I've been worried that AI capabilities evals aren't sufficiently useful to policymakers lately, and that worry has been fueled by some of these excellent pieces: { Anton Lecht, on political economy problems for AI evals (www.antonleicht.me/writing/evals),

antonleicht.meProblems in the AI Eval Political Economy — Anton LeichtEvaluations of new AI models’ capabilities and risks are an important cornerstone of safety-focused AI policy. Currently, their future as part of the policy platform faces peril for four reasons: Enta...

0 reposts0 likes

Henry🔸
@henryjosephson.com

Gwern on benchmarking creativity (gwern.net/creative-ben...), Holden Karnofsky on tripwires (carnegieendowment.org/research/202..., carnegieendowment.org/research/202...), UK AISI evals lessons (www.aisi.gov.uk/work/early-l...), AI LabWatch (ailabwatch.org/blog/model-e...), }

gwern.netTowards Benchmarking LLM Diversity & CreativityDiscussion of possible tasks to measure LLM capabilities in soft ‘creative’ tasks like brainstorming or editing, to quantify failures in creative writing domains.