Giles Thomas

Giles Thomas

@gilesthomas.com

Building LLMs, learning in public / Founded and led @PythonAnywhere.com / PSF Fellow / Writing at gilesthomas.com

227 followers 67 following143 posts

Pinned

@gilesthomas.com

Interesting. By replacing the embedding layer with a low-rank bottleneck, I managed to *improve* my model's test loss while reducing parameter count by from 163M to 131M. Only two runs, so far from proven, but it's a surprising result! www.gilesthomas.com/2026/10/low-...

gilesthomas.comFun with low-rank vocab matrices (and a bonus test loss reduction?)I was told you can swap an LLM's embeddings and output head for low-rank factors without hurting the loss much. Was that true? Not in my tests. But doing just the embeddings seemed to improve it!

@gilesthomas.com

A (relatively) quick post on RMSNorm: how it works, and does it really speed up training for minimal loss in performance when compared to LayerNorm? In my tests on a GPT-2-like model, loss stayed the same (within the noise) but training speed hardly changed. www.gilesthomas.com/2026/10/rmsn...

gilesthomas.comRMSNormThe original RMSNorm paper found that it was faster than LayerNorm, with little to no effect on training loss. Would that hold up for my models?

@gilesthomas.com

The real winner was using built-in PyTorch normalisation classes -- both nn.LayerNorm or nn.RMSNorm are *way* faster than hand-rolled versions, and weirdly (in these tests) got better loss too -- though I think that must be chance.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Next step in trying to work out why GPT-2 small beats my own similar models: is it the data quality? Maybe... www.gilesthomas.com/2026/10/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine? Part five: data qualityDoes improving the training data quality help me close the gap between my models and the original GPT-2 small on my IFT eval?

@gilesthomas.com

The first in what will probably be an occasional series on my blog: Why do OpenAI's GPT-2 weights beat mine? www.gilesthomas.com/2026/07/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine?The original GPT-2 small weights are much better than my own models at a specific instruction-following task. Time to look into why!

View full thread

@gilesthomas.com

The next step in the saga of why OpenAI's GPT-2 weights beat mine on instruction fine-tuning: what about dropout? www.gilesthomas.com/2026/08/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine? Part four: digging into dropoutMight whether or not I use dropout while doing multiple epochs of fine-tuning hold the key to the variation in the results?

@gilesthomas.com

Next step in trying to work out why GPT-2 small beats my own similar models: is it the data quality? Maybe... www.gilesthomas.com/2026/10/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine? Part five: data qualityDoes improving the training data quality help me close the gap between my models and the original GPT-2 small on my IFT eval?

@gilesthomas.com

I extended the GPT-2-style code from @rasbt's "Build a Large Language Model (from Scratch)" so that it was a 6-expert (2 active) mixture-of-experts, and trained it from scratch over 8 days. It worked well! Full writeup with maths and code at www.gilesthomas.com/2026/09/gpt-...

gilesthomas.comExtending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090How mixture-of-experts LLMs work, both in theory and in real working code based on 'Build a Large Language Model (from Scratch)'.

@gilesthomas.com

A new blog post: Putting my JAX-trained models on the Hugging Face Hub www.gilesthomas.com/2026/09/jax-...

gilesthomas.comPutting my JAX-trained models on the Hugging Face HubI'd not been putting my JAX models on the Hugging Face Hub because it seemed hard, but then realised I could just upload the tensors converted to PyTorch

@gilesthomas.com

D2 seems pretty nifty for diagramming. Just added support for it to my blog's static site generator: www.gilesthomas.com/2026/08/addi...

gilesthomas.comAdding diagrams to my static site generator with D2I wanted to have an easier way to put diagrams on this blog. D2 seems to work pretty well!

@gilesthomas.com

By switching from a hand-rolled GELU to PyTorch's built-in one, I improved my LLM training speed -- and much more than I expected, from 21,000 tokens/second to 25,000! www.gilesthomas.com/2026/08/buil...

gilesthomas.comUse the built-in GELU, don't roll your own!Switching from the hand-rolled GELU module I'd been using to PyTorch's built-in one unsurprisingly made training models faster -- but I was surprised to see it was 20% faster.

@gilesthomas.com

I'd trained some models on 40 tokens per parameter. The Chinchilla paper says that doing that is suboptimal. I wanted to see if that was correct for my training setup -- and it looks like it was :-) www.gilesthomas.com/2026/08/chin...

gilesthomas.comA quick(ish) Chinchilla checkHaving overtrained two models, I decided to use an ~equivalent number of FLOPs to train Chinchilla-optimal models. Would the rule hold up?

@gilesthomas.com

I thought I'd put together a quick post on how I use AI when writing my blog: AIs identify problems and then I fix them myself. www.gilesthomas.com/2026/07/ai-use

gilesthomas.comHow I use AI on this blogA point-in-time snapshot of how I'm currently using AI to help with my experiments, and with blogging about them

@gilesthomas.com

The first in what will probably be an occasional series on my blog: Why do OpenAI's GPT-2 weights beat mine? www.gilesthomas.com/2026/07/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine?The original GPT-2 small weights are much better than my own models at a specific instruction-following task. Time to look into why!

@gilesthomas.com

So, the first step in solving the mystery was to fix a bug in the eval! The mystery remains, though -- OpenAI's GPT-2 models are still better at instruction-following than mine. www.gilesthomas.com/2026/07/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine? Part two: the bugfixBefore I get started on my attempts to make my models as good as the original GPT-2 ones at instruction-following, there was a bug I needed to fix.

@gilesthomas.com

With the eval bug fixed, it was time to test something. If I train for twice as long -- by Chinchilla standards, overtraining -- do my models get better? The answer is, annoyingly, "...maybe?" www.gilesthomas.com/2026/07/why-...

gilesthomas.comWhy do OpenAI's GPT-2 weights beat mine? Part three: testing overtrainingDoes deliberately overtraining my own GPT-2 small style models match the original models' instruction-following abilities?

@gilesthomas.com

A friend asked me how many tps I could get with Qwen 3.6 35B MoE on my RTX 3090. I may have overthought the question, but maybe the results are useful for other people? www.gilesthomas.com/2026/07/benc...

gilesthomas.comBenchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090A friend asked me how many tokens per second I could get on Qwen 3.6 35B MoE on an RTX 3090. I probably dug into it more than he expected.

@gilesthomas.com

With no weight tying, the token embeddings and the output head alone make up almost half of a GPT-2-small sized model. That makes sense, but it's kind of unintuitive! I asked GPT 5.6 Sol to create a visualiser: www.gilesthomas.com/post-assets/...

gilesthomas.comGPT-2 Parameter CounterExplore how embedding width and layer count shape a GPT-2 model's parameter count.

@gilesthomas.com

I'm repurposing an old PC as a dedicated LLM training box, and blogging the process. Here's part 1, featuring an almost-melted CPU and an accidental 11-day LLM training run: www.gilesthomas.com/2026/07/popp...

gilesthomas.compoppy the training box, part 1: the beginningsRepurposing an old SFF PC as a dedicated LLM training box: a second-hand RTX 3090, an accidental 11-day train on a GTX 1660, and one dead CPU fan.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

And that's it! After 18 months, the capstone in my LLM from scratch journey. To test my knowledge, I built GPT-2 small in JAX, using just my notes. It worked really well, and I found an interesting way to put it together bit-by-bit, watching the loss go down. www.gilesthomas.com/2026/07/llm-...

gilesthomas.comWriting an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Coming to the end of my LLM from scratch journey... I'm building a GPT-2-small in JAX, and decided to build the training loop first, then build the model step-by-step. Here's how I build the training loop. www.gilesthomas.com/2026/06/llm-...

gilesthomas.comWriting an LLM from scratch, part 34a -- building a JAX training loop for an LLM training runI'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.

@gilesthomas.com

A bit of a debugging journey with JAX, Flax and NNX: www.gilesthomas.com/2026/06/hash...

gilesthomas.comFlax debugging: making a hash of thingsA quick way to help with debugging plumbing issues in JAX training loops.

@gilesthomas.com

Switching from a power-hungry Marvell-based SFP+ 10GBASE-T module to a cooler Broadcom-based one: some oddities, including a mendacious EEPROM. www.gilesthomas.com/2026/06/10g-...

gilesthomas.com10Gb/s Ethernet: switching to a Broadcom SFP+ moduleAs predicted by several people, one of my Marvell-based SFP+ modules had overheating problems. I've switched it over to a Broadcom-based one, and it looks better, though I can't be certain.

@gilesthomas.com

Hit some interesting memory management weirdness in JAX: arrays can move around between devices when you don't expect it, if you don't commit them to one explicitly: www.gilesthomas.com/2026/06/jax-...

gilesthomas.comJAX: commitment issuesJAX memory management is tricky; data can be moved around unexpectedly if you don't make sure it's committed to a device.

@gilesthomas.com

More notes on working with JAX: backends and devices. www.gilesthomas.com/2026/06/jax-...

gilesthomas.comJAX backends and devicesSome basic notes on how to get JAX to load data to a particular device.

@gilesthomas.com

Getting JAX/Flax and Safetensors to play nicely was a bit fiddly -- wrote it up here: www.gilesthomas.com/2026/06/flax...

gilesthomas.comUsing Safetensors with FlaxUsing Safetensors to save Flax models isn't all that hard, so long as you know the trick.

@gilesthomas.com

I've spent some time learning JAX over the last month, and I have Thoughts: www.gilesthomas.com/2026/05/on-f...

gilesthomas.comOn first looking into JAXSome (well, quite a lot of) initial thoughts about JAX, having spent a short time playing with it.

@gilesthomas.com

WiFi has made home wired networking boring for the last 20 years or so, I think, but 10Gb/s "fixes" that :-) I've had a bit of fun over the last two weeks doing an upgrade, here's part 1 of a two-part writeup: www.gilesthomas.com/2026/04/10g-...

gilesthomas.com10Gb Ethernet: what I had to (re)learnWired networking in the home and small offices has been pretty stagnant for ages, so upgrading my home from 2.5Gb ethernet to 10Gb meant I had to (re)learn a few things.

@gilesthomas.com

...and here's part 2: what I actually did. Turns out you could use an SFP+ 10GBASE-T module to make a (very small) cup of tea: www.gilesthomas.com/2026/04/10g-...

gilesthomas.com10Gb/s Ethernet: what I actually did to get it working in my homeI already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.

@gilesthomas.com

Using a mini-heatsink designed for a Raspberry Pi helps cool the 10GBASE-T SFP+ module! But, unfortunately, not very much. www.gilesthomas.com/2026/05/10g-...

gilesthomas.com10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ moduleA quick update to show the (smallish) benefit of adding mini-heatsinks to a 10GBASE-T SFP+ module

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

...and here's part 2: what I actually did. Turns out you could use an SFP+ 10GBASE-T module to make a (very small) cup of tea: www.gilesthomas.com/2026/04/10g-...

gilesthomas.com10Gb/s Ethernet: what I actually did to get it working in my homeI already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

The appendices in "Build an LLM (from Scratch)" have a lot of really useful stuff. Should I have read them before heading off on my own training runs? I think no -- it would have saved time, but by trying, failing, and trying again, I learned more. YMMV! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 33 -- what I learned from finally getting round to the appendicesI've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Time to wrap up my "Interventions" series: www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32m -- Interventions: conclusionWrapping up my 'Interventions' mini-series: what I've learned, what I've achieved, and what's next!

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Some interesting results with instruction fine-tuning. In general you'd expect that a model with lower loss on your test set would fine-tune to do better instruction-following. But the correlation doesn't seem as close as I would have thought: www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning resultsI wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!

@gilesthomas.com

I had 57 checkpoints from my last local LLM training run on my disk, and thought it would be interesting to use them to show how the model got more coherent over time. By 1/3 of the way through, it was surprisingly solid! www.gilesthomas.com/2026/04/how-...

gilesthomas.comHow an LLM becomes more coherent as we train itAs an LLM is trained, it gradually learns how to generate increasingly coherent text. Here's an example.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Training a new GPT-2-small-style base model on my RTX 3090 in 40 hours: gradient accumulation, along with some of the other interventions I've experimented with, got me almost (but not quite) all the way to the quality of the original OpenAI model! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32k -- Interventions: training a better model locally with gradient accumulationHaving worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

After spending two months trying out different interventions on my GPT-2-style model, it was time to try stacking them up. Interesting results! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloudNow that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

I wanted to see whether my results when testing interventions to my GPT-2-style training loop were signal or noise. The results were promising! www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32i -- Interventions: what is in the noise?How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

My final intervention test, in which I discover that there is such a thing as a free lunch, and it's called AMP. www.gilesthomas.com/2026/04/llm-...

gilesthomas.comWriting an LLM from scratch, part 32h -- Interventions: full fat float32I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Weight tying, by contrast with weight decay, was actually really easy! It didn't help, though :-( www.gilesthomas.com/2026/03/llm-...

gilesthomas.comWriting an LLM from scratch, part 32g -- Interventions: weight tyingWeight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?

Giles Thomas reposted

@gilesthomas.com

Replying to @gilesthomas.com

Weight decay is conceptually simpler than I worried it might be, but still pretty fiddly to get right... www.gilesthomas.com/2026/03/llm-...

gilesthomas.comWriting an LLM from scratch, part 32f -- Interventions: weight decayWhat is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?