Post

1/6 What does MSA Pairformer learns about evolutionary relationships? @yoakiyama.bsky.social et al achieved impressive contact & variant-fitness predictions w a fraction of ESM2's parameters; we wanted to better understand the mechanism arcadia-science.github.io/2025-phyloge... 🧪🧬 🖥️

2/6 Their key innovation is query-biased attention, where the model weights MSA sequences by relevance to the query. We tested whether "relevance" maps onto phylogenetic distance by correlating sequence weights with evolutionary distances across thousands of protein families.

3/6 The model consistently upweights evolutionarily close sequences and downweights distant ones, despite never being trained on this. There's also a division of labor across layers: layer 11 acts as a strong phylogenetic filter, while others contribute more modestly.

5/6 We traced this to layers 5-11, which integrate sequence weights blindly, regardless of whether they help or hurt. When they hurt prediction, we could progressively rescue performance by shuffling weights at earlier layers, saving the model from its own bad intuition.

6/6 As the field shifts toward MSA-based architectures, the quality of input alignments becomes paramount. MSA Pairformer encodes evolutionary signal well, but future architectures might benefit from learning when to trust that signal.