Post

Core skills - they identify some key areas open post-training models struggle with and target those for improving, picking datasets in those areas for evaluation. Notably, they avoid looking at scores from the unseen set during development to assess if their choices might have led to overfitting.

docs.google.comBig Instruction Compilation

Data sources - they curate from public datasets based on three criteria - diversity (for generalization), targeting their identified core skills and clear licensing/provenance for proper usage. Datasets they considered: docs.google.com/spreadsheets...

Data sources - they combat mode collapse using persona-driven synthetic data generation, where unique personas (e.g., "ML researcher focused on neural networks") guide LLMs to create diverse training data for specific skills like precise instruction following, math, and coding.

Data decontamination - n-gram matching works better than embedding methods in terms of precision - 8-gram matching only on user-turns (since completions are LLM generated) - remove training sets that have more than 2% overlap with any evaluation set

SFT - for the data, train models on skill specific data mixtures, keep the mixtures that lead to the best performance on those skills. Later, combine the mixtures and do decontamination/downsampling of larger datasets.

24/11/24, 6:04 pm0 reposts0 likes

SFT - a couple of data ablations: - Adding safety data didn't negatively affect other skills. - Measured the removal of math data on GSM8K and MATH, confirming that the scores drop. - Measured the the effectiveness of WildChat (diverse chat data) and Persona (skill based data)

SFT - using a better pretrained model (Qwen 2.5) results in better scores for GSM8K and MATH. Kinda surprising since the final model is still Llama 3.1, wonder if they evaluated Qwen 2.5 on other datasets as well and if the performance was not good enough to be the base model?

SFT - performance varies greatly depending on the random seed. They tried model soups (averaging weights of multiple models using github.com/arcee-ai/mer...), but decided to choose a single best model instead.