Post

Google managed to make the mobile versions of Gemma 4 E2B and E4B remarkably small, I'll have to look into how they manage this

Google for Developers@developers.google.com

Gemma 4 quantization-aware training (QAT) models are now available, bringing AI performance directly to edge devices and consumer GPUs. These checkpoints are optimized with quantization-aware training to dramatically reduce memory requirements and unlock high-speed local inference. 🧵

5/6/26, 6:03 pm1 repost36 likes

I’m expecting that the new Siri is going to be powered by Gemma 4 as a base. Google has some good reasons of their own to develop on-device models of course. Still, I imagine we’ll see something like a Gemma-based on-device model, with. A Gemini-based fallback in the cloud for tougher queries.

Tim Duffy

@timfduffy.com

6 June

I'm still pretty skeptical of edge inference, though Gemma is def bringing it closer with this release. I just don't see that much advantage vs a cheap model like Gemini 3 flash. Token volume for Siri will probably be fairly small, so price doesn't seem like a crucial factor. And I don't think...

Tim Duffy

@timfduffy.com

6 June

local processing is likely to be faster either. It probably takes at least a second to load the model, and TPS is going to be lower locally.

Ben Carson

@mensmachina.com

6 June

I think it’s more ideology than efficiency in Apple’s case. Given so much of their brand positioning is on privacy, and on-device processing I’m expecting to continue in that vein. It’ll be interesting to see if there’s any tricks that Apple can play to manage memory and model latency.

Tim Duffy

@timfduffy.com

5 June

They give some reasons in this post but I don't understand all of them or how it gets them so much smaller than the 4-bit version

Apex

@apex.atproto.ceo

5 June

it's gonna be those last two points that save the most memory and mainly by making the model a little dumber and more forgetful

Tim Duffy

@timfduffy.com

5 June

Yeah I think you're right. I was confused about the last one in particular at first since the config.json listed the same vocab size, but to get it that small it must be partly about reducing vocab size since the per-layer embeddings are such a large proportion of the total size.