Post

Tim Duffy
@timfduffy.com

Google managed to make the mobile versions of Gemma 4 E2B and E4B remarkably small, I'll have to look into how they manage this

Google for Developers@developers.google.com

Gemma 4 quantization-aware training (QAT) models are now available, bringing AI performance directly to edge devices and consumer GPUs. These checkpoints are optimized with quantization-aware training to dramatically reduce memory requirements and unlock high-speed local inference. 🧵

Tim Duffy
@timfduffy.com

They give some reasons in this post but I don't understand all of them or how it gets them so much smaller than the 4-bit version

0 reposts12 likes

Apex
@apex.atproto.ceo

it's gonna be those last two points that save the most memory and mainly by making the model a little dumber and more forgetful

Tim Duffy
@timfduffy.com

Yeah I think you're right. I was confused about the last one in particular at first since the config.json listed the same vocab size, but to get it that small it must be partly about reducing vocab size since the per-layer embeddings are such a large proportion of the total size.