Post

A new paper dropped from DeepMind: Deliberation in Latent Space via Differentiable Cache Augmentation The trouble is, it's not very readable. I tried making a thread here, but it got far too long, so it's a blog now: timkellogg.me/blog/2024/12...

Thanks for sharing your blog post! But I also would so be up for a long thread form of this XD When you mentioned working in the latent space, I instantly thought of this late 2023 paper by Google Research arxiv.org/abs/2310.04475

interesting, i’ve heard of CAVs, but they always seemed more of a gray box approach. it seems like you could train a model to translate an embedding back into text. doesn’t seem like it would be too hard. also seems inevitable if we go all in on latent space

But translating embedding back to text seems to me not necessarily straightforward. Where I've seen success is when comparisons are done with a 'well-understood' embedding. I can see conversion to text being difficult in a similar way that there doesn't exist perfect translation between languages

Reminded me of this paper: arxiv.org/abs/2310.06816