Post

Pretraining the experts on different tasks sounds good, because you could freeze them. Then, the router would get gradients that say "this model would've done a better job for this task, so pick it next time".

However, that is also partly true for random init, and even if it wasn't it's outweighed by allowing the model to designate experts for abstract tasks, so every expert doesn't need to know everything.