Post

Chip Huyen
@chiphuyen.bsky.social

2. OpenAI’s best practices for finetuning While this guide focuses on GPT-3, many techniques are applicable to finetuning in general. It explains how finetuning works, how to prepare training data, how to pick hyperparameters, and common finetuning mistakes: docs.google.com/document/d/1...

docs.google.com[PUBLIC] Best practices for fine-tuning GPT-3 to classify textThis document is a draft of a guide that will be added to a future revision of the OpenAI documentation. If you have any feedback, feel free to let us know. One note: this doc shares metrics for text...
Chip Huyen
@chiphuyen.bsky.social

3. Llama 3 paper The section on post-training data is a gold mine! It details different techniques they used to generate 2.7M examples for instruction finetuning. It also covers synthetic data verification! arxiv.org/abs/2407.21783

arxiv.orgThe Llama 3 Herd of ModelsModern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support ...
Chip Huyen
@chiphuyen.bsky.social

4. Efficiently Scaling Transformer Inference (Pope et al., 2022) An amazing paper about inference optimization for transformers. It provides a guideline to optimize for different aspects, e.g. lowest possible latency, highest possible throughput, or longest context length: arxiv.org/abs/2211.05102

arxiv.orgEfficiently Scaling Transformer InferenceWe study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better ...
Chip Huyen
@chiphuyen.bsky.social

5. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (Lu et al., 2023) A cool study on LLM planners, how they use tools, and their failure modes. An interesting finding is that different LLMs have different tool preferences: arxiv.org/abs/2304.09842

arxiv.orgChameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsLarge language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they...

No replies yet.