Post

1/6 What does MSA Pairformer learns about evolutionary relationships? @yoakiyama.bsky.social et al achieved impressive contact & variant-fitness predictions w a fraction of ESM2's parameters; we wanted to better understand the mechanism arcadia-science.github.io/2025-phyloge... 🧪🧬 🖥️

2/6 Their key innovation is query-biased attention, where the model weights MSA sequences by relevance to the query. We tested whether "relevance" maps onto phylogenetic distance by correlating sequence weights with evolutionary distances across thousands of protein families.

4/6 But surprisingly, encoding phylogeny ≠ leveraging it. Comparing learned weights against uniform averaging for contact prediction, the aggregate improvement was marginal. Per-MSA, uniform averaging outperformed learned weights about half the time, sometimes by a large margin.

5/6 We traced this to layers 5-11, which integrate sequence weights blindly, regardless of whether they help or hurt. When they hurt prediction, we could progressively rescue performance by shuffling weights at earlier layers, saving the model from its own bad intuition.

6/6 As the field shifts toward MSA-based architectures, the quality of input alignments becomes paramount. MSA Pairformer encodes evolutionary signal well, but future architectures might benefit from learning when to trust that signal.