Post

A new paper dropped from DeepMind: Deliberation in Latent Space via Differentiable Cache Augmentation The trouble is, it's not very readable. I tried making a thread here, but it got far too long, so it's a blog now: timkellogg.me/blog/2024/12...

timkellogg.meExplainer: Latent Space Experts - Tim Kellogg

mr. TIM

@timkellogg.me

24 Dec 2024

The big notes here are: * Use an "expert" LLM to augment a generalist * Yes, it's like RAG, but it doesn't replace RAG

jake

@yetanotheruseless.com

It doesn't really look like RAG at all, is it? It looks much more like a different way to handle internal thoughts in embedding space rather than forcing them to pick concrete tokens to give it space to calculate...

24/12/24, 7:11 pm0 reposts0 likes

mr. TIM

@timkellogg.me

24 Dec 2024

a lot of people see RAG as a cheaper way to get a domain specific version of a model. atm RAG is actually a reasonable approach, but this might be better

jake

@yetanotheruseless.com

24 Dec 2024

Yeah RAG can bring in domain grounding, and LoRA fine tuning can shift to domain styling/formatting. This seems more general than that: the main experiments I saw in the paper were training the coprocessor with the same pretraining as the base model. “To learn to think” more than domain adapt.