Post

Dion Almaer
@almaer.com

Gemini Diffusion! I love a good zig and zag. We used to be in a world where image generation used diffusion techniques (from noise), and text used autoregresion (next token). ChatGPT added 4o image generation that went the other way and used autoregression …

0 reposts3 likes

Dion Almaer
@almaer.com

And today we see the zag. We have another tool in the toolbox, diffusion models for text. This isn’t new, and has been talked about for some time. Unlike next token, diffusion by its nature is a good fit for generating text in a non-sequential manner…

Dion Almaer
@almaer.com

If there are gaps that you want to fill in, this could have better results. It also allows for more parallelism, and doesn’t fall into the trap of being stuck on a path due to the words already generated. I’m keen to see how this works for codegen!!