Post

Core skills - they identify some key areas open post-training models struggle with and target those for improving, picking datasets in those areas for evaluation. Notably, they avoid looking at scores from the unseen set during development to assess if their choices might have led to overfitting.

Data sources - they curate from public datasets based on three criteria - diversity (for generalization), targeting their identified core skills and clear licensing/provenance for proper usage. Datasets they considered: docs.google.com/spreadsheets...

Data sources - they combat mode collapse using persona-driven synthetic data generation, where unique personas (e.g., "ML researcher focused on neural networks") guide LLMs to create diverse training data for specific skills like precise instruction following, math, and coding.