Post

@gilesthomas.com

Back when I started messing with LLMs, it looked to me like you could get reasonably OK results for chat applications without instruction fine-tuning. So before getting into Chapter 7 of "Build an LLM (from scratch)", I decided to see if that was really true: www.gilesthomas.com/2025/10/llm-...

gilesthomas.comWriting an LLM from scratch, part 24 -- the transcript hackBack when I started playing with LLMs, I found that you could build a (very basic) chatbot with a base model -- no instruction fine-tuning at all! Does that work with GPT-2?

@gilesthomas.com

Back on track with chapter 7 of "Build an LLM (from scratch)": notes on instruction fine-tuning of our GPT-2 model: www.gilesthomas.com/2025/10/llm-...

gilesthomas.comWriting an LLM from scratch, part 25 -- instruction fine-tuningSome notes on the first part of chapter 7 of 'Build an LLM (from scratch)': instruction fine-tuning

@gilesthomas.com

The end of the beginning... running evals on our model using Llama 3 is the last part of the main body of @sebastianraschka.com's "Build an LLM (from scratch)". Here's my writeup: www.gilesthomas.com/2025/11/llm-...

gilesthomas.comWriting an LLM from scratch, part 26 -- evaluating the fine-tuned modelComing to the end of 'Build an LLM (from scratch)'! We evaluate the quality of the responses our model produces.

@gilesthomas.com

So, what's left to do in my series on building an LLM from scratch? And what follow-up series should I work on? Some musings: www.gilesthomas.com/2025/11/llm-...

gilesthomas.comWriting an LLM from scratch, part 27 -- what's left, and what's next?Having finished the main body of 'Build an LLM (from scratch)', it's time to think about what I need to do to treat this project as fully done

@gilesthomas.com

I managed to train my own base model from scratch on an RTX 3090! Very detailed notes here: www.gilesthomas.com/2025/12/llm-...

gilesthomas.comWriting an LLM from scratch, part 28 -- training a base model from scratch on an RTX 3090I felt like it should be possible to train a GPT-2 small level model on my own hardware using modern tools and open datasets from scratch. It was!

@gilesthomas.com

Having trained a GPT-2 scale base model from scratch in 48 hours locally, I wanted to see if I could do the same faster and at a reasonable cost in the cloud. I could! www.gilesthomas.com/2026/01/llm-...

gilesthomas.comWriting an LLM from scratch, part 29 -- using DistributedDataParallel to train a base model from scratch in the cloudHaving trained a base model from scratch on my own machine over 48 hours, I wanted to make it faster by training with multiple GPUs in the cloud.

@gilesthomas.com

I wanted to dig into why the results I got on instruction fine-tuning for each of my models didn't seem to match up well with the loss on the test set. Got some interesting results: www.gilesthomas.com/2026/01/2026...

gilesthomas.comWriting an LLM from scratch, part 30 -- digging into the LLM-as-a-judge resultsI was unhappy with the LLM-as-a-judge instruction fine-tuning results I got when comparing my various base models. Could I make them any better?

@gilesthomas.com

I thought it would be good to share the base models I've been training on Hugging Face, and now they are :-) www.gilesthomas.com/2026/01/llm-...

gilesthomas.comWriting an LLM from scratch, part 31 -- the models are now on Hugging FaceI've trained seven models using the GPT-2 architecture: let's share them on Hugging Face!

@gilesthomas.com

Back to my LLM from scratch series. I want to train the *best* GPT-2-style model that I can locally in two days, and there are various levers to pull. Working out which ones work means I need a baseline for comparison. www.gilesthomas.com/2026/02/llm-...

gilesthomas.comWriting an LLM from scratch, part 32a -- Interventions: training a baseline modelI want to try a bunch of interventions like gradient clipping and removing dropout to see if my models get better. I need a baseline train without them so that I can be sure of the results.

@gilesthomas.com

First "intervention" test: does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected. www.gilesthomas.com/2026/02/llm-...

gilesthomas.comWriting an LLM from scratch, part 32b -- Interventions: gradient clippingDoes adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected.

@gilesthomas.com

Does removing dropout improve our baseline model's test loss? Yes, absolutely, and much more than gradient clipping did. www.gilesthomas.com/2026/02/llm-...

gilesthomas.comWriting an LLM from scratch, part 32c -- Interventions: removing dropoutDoes removing dropout improve our baseline model's test loss? Yes, absolutely, and even more than gradient clipping did.

@gilesthomas.com

Now this was a surprise. QKV bias is not meant to be useful -- but with my GPT-2 small model, it looks like it is! www.gilesthomas.com/2026/02/llm-...

gilesthomas.comWriting an LLM from scratch, part 32d -- Interventions: adding attention biasHaving bias terms for the query, key, and value attention weights is apparently no longer used because it doesn't help. Let's check that it really doesn't for our model!

@gilesthomas.com

Learning rates for LLMs turned out to be a deep topic, but making some tweaks certainly seems to help my base model train: www.gilesthomas.com/2026/03/llm-...

gilesthomas.comWriting an LLM from scratch, part 32e -- Interventions: the learning rateThe learning rate is an essential hyperparameter for training models with gradient descent, but what actual values should you use, and what does it mean to schedule it?

@gilesthomas.com

Weight decay is conceptually simpler than I worried it might be, but still pretty fiddly to get right... www.gilesthomas.com/2026/03/llm-...

gilesthomas.comWriting an LLM from scratch, part 32f -- Interventions: weight decayWhat is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?

@gilesthomas.com

Weight tying, by contrast with weight decay, was actually really easy! It didn't help, though :-( www.gilesthomas.com/2026/03/llm-...

gilesthomas.comWriting an LLM from scratch, part 32g -- Interventions: weight tyingWeight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?

@gilesthomas.com

My final intervention test, in which I discover that there is such a thing as a free lunch, and it's called AMP. www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32h -- Interventions: full fat float32I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...

@gilesthomas.com

I wanted to see whether my results when testing interventions to my GPT-2-style training loop were signal or noise. The results were promising! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32i -- Interventions: what is in the noise?How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.

@gilesthomas.com

After spending two months trying out different interventions on my GPT-2-style model, it was time to try stacking them up. Interesting results! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloudNow that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?

@gilesthomas.com

Training a new GPT-2-small-style base model on my RTX 3090 in 40 hours: gradient accumulation, along with some of the other interventions I've experimented with, got me almost (but not quite) all the way to the quality of the original OpenAI model! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32k -- Interventions: training a better model locally with gradient accumulationHaving worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...

@gilesthomas.com

Some interesting results with instruction fine-tuning. In general you'd expect that a model with lower loss on your test set would fine-tune to do better instruction-following. But the correlation doesn't seem as close as I would have thought: www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning resultsI wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!

@gilesthomas.com

Time to wrap up my "Interventions" series: www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32m -- Interventions: conclusionWrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!

21/4/26, 5:09 pm1 repost0 likes

@gilesthomas.com

The appendices in "Build an LLM (from Scratch)" have a lot of really useful stuff. Should I have read them before heading off on my own training runs? I think no -- it would have saved time, but by trying, failing, and trying again, I learned more. YMMV! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 33 -- what I learned from finally getting round to the appendicesI've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.

@gilesthomas.com

Coming to the end of my LLM from scratch journey... I'm building a GPT-2-small in JAX, and decided to build the training loop first, then build the model step-by-step. Here's how I build the training loop. www.gilesthomas.com/2026/06/llm-...

gilesthomas.comWriting an LLM from scratch, part 34a -- building a JAX training loop for an LLM training runI'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.

@gilesthomas.com

And that's it! After 18 months, the capstone in my LLM from scratch journey. To test my knowledge, I built GPT-2 small in JAX, using just my notes. It worked really well, and I found an interesting way to put it together bit-by-bit, watching the loss go down. www.gilesthomas.com/2026/07/llm-...

gilesthomas.comWriting an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.