Post

Pretraining the experts on different tasks sounds good, because you could freeze them. Then, the router would get gradients that say "this model would've done a better job for this task, so pick it next time".

Noa Roggendorff

@sly.sh

However, that is also partly true for random init, and even if it wasn't it's outweighed by allowing the model to designate experts for abstract tasks, so every expert doesn't need to know everything.

7/10/26, 3:46 am0 reposts0 likes

Noa Roggendorff

@sly.sh

Although, the gate gradients are probably completely pointless for a random-init expert, as all experts should be equally bad.