Post

Giles Thomas
@gilesthomas.com

What better use of a public holiday than learning how multi-head attention works? More notes from reading through @sebastianraschka.com's "Build a Large Language Model (from Scratch)": www.gilesthomas.com/2025/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 12 -- multi-head attentionFinally getting to the end of chapter 3 of Raschka’s LLM book! This time it’s multi-head attention: what it is, how it works, and why the code does what it does.
Giles Thomas
@gilesthomas.com

Moving onwards with @sebastianraschka.com's "Build an LLM (from scratch)". The last step in an LLM, going from context vectors to next-word prediction, looks insanely simple. How can it possibly work? www.gilesthomas.com/2025/05/llm-...

gilesthomas.comWriting an LLM from scratch, part 15 -- from context vectors to logits; or, can it really be that simple?!The way we get from context vectors to next-word prediction turns out to be simpler than I imagined -- but understanding why it works took a bit of thought.
Giles Thomas
@gilesthomas.com

I wanted to do a "what I've learned so far" post to wrap up my notes on Chapter 4 of @sebastianraschka.com's "Build an LLM (from scratch)" but when it got to 6,000 words I suspected it was getting a bit long. So here's a review of upcoming attractions: www.gilesthomas.com/2025/08/llm-...

gilesthomas.comWriting an LLM from scratch, part 19 -- wrapping up Chapter 4A state-of-play update after finishing Chapter 4 of 'Build a Large Language Model from Scratch', with a roadmap of what’s coming next
Giles Thomas
@gilesthomas.com

Part 22 is live: we finally train the LLM :-) Following @sebastianraschka.com's book, we train on Edith Wharton, then swap in GPT-2 (124M) weights for comparison. Notes on seeding, AdamW, temperature and top-k. www.gilesthomas.com/2025/10/llm-...

gilesthomas.comWriting an LLM from scratch, part 22 -- finally training our LLM!Finally, we train an LLM! The final part of Chapter 5 of Build an LLM (from Scratch) runs the model on real text, then loads OpenAI’s GPT-2 weights for comparison.
Giles Thomas
@gilesthomas.com

Back when I started messing with LLMs, it looked to me like you could get reasonably OK results for chat applications without instruction fine-tuning. So before getting into Chapter 7 of "Build an LLM (from scratch)", I decided to see if that was really true: www.gilesthomas.com/2025/10/llm-...

gilesthomas.comWriting an LLM from scratch, part 24 -- the transcript hackBack when I started playing with LLMs, I found that you could build a (very basic) chatbot with a base model -- no instruction fine-tuning at all! Does that work with GPT-2?
Giles Thomas
@gilesthomas.com

Having trained a GPT-2 scale base model from scratch in 48 hours locally, I wanted to see if I could do the same faster and at a reasonable cost in the cloud. I could! www.gilesthomas.com/2026/01/llm-...

gilesthomas.comWriting an LLM from scratch, part 29 -- using DistributedDataParallel to train a base model from scratch in the cloudHaving trained a base model from scratch on my own machine over 48 hours, I wanted to make it faster by training with multiple GPUs in the cloud.