Saez-Rodriguez Group

20 Nov 2024

We have created a starterpack to make it easier to follow current members and alumni from our lab, check it out 👇 (DM us if we forgot about you) Also juliosaezrod.bsky.social has recently joined Bluesky go.bsky.app/5ZtPHPz

This month we welcomed two new PhD students and a new postdoc to the lab, Stanislaus Stein, Amber Fijn and Alicja Wojciechowska. 🎉

Happy to see the peer-reviewed version of SpaCEy now out: 📄 Published paper: www.nature.com/articles/s41... 💻 Code: github.com/saezlab/SpaCEy

Original preprint: www.biorxiv.org/content/10.6...

Many thanks to everyone who contributed to the work and to the reviewers for helping us strengthen the study. #SpatialOmics #SpatialBiology #GraphNeuralNetworks #ExplainableAI #ComputationalBiology #CancerResearch

Happy to see the peer-reviewed version of SpaCEy now out: 📄 Published paper: www.nature.com/articles/s41... 💻 Code: github.com/saezlab/SpaCEy

- We added systematic scalability analyses across increasing numbers of samples and molecular markers, evaluating both computational time and memory requirements.

This study was led by Ahmet Sureyya Rifaioglu with the help of Ahmet Sarigun and Deniz Germen and Egle Helene Ervin. The work was carried out in collaboration with the Bodenmiller Lab. The study was co-supervised by Julio Saez-Rodriguez and Jovan Tanevski.

Happy to see the peer-reviewed version of SpaCEy now out: 📄 Published paper: www.nature.com/articles/s41... 💻 Code: github.com/saezlab/SpaCEy

- We strengthened the survival analyses across two independent breast cancer spatial proteomics cohorts, including analyses within the HR+HER2− subtype to assess clinically relevant heterogeneity beyond established subtype classifications.

- We further investigated the spatial context of SpaCEy-important regions, connecting the model explanations to tumour, peritumoural, and extratumoural tissue compartments.

Happy to see the peer-reviewed version of SpaCEy now out: 📄 Published paper: www.nature.com/articles/s41... 💻 Code: github.com/saezlab/SpaCEy

Following peer review, the study has been substantially extended with additional analyses.

- We expanded the benchmarking against both spatial and non-spatial approaches, further evaluating the contribution of spatial tissue organization to clinical outcome prediction.

Happy to see the peer-reviewed version of SpaCEy now out: 📄 Published paper: www.nature.com/articles/s41... 💻 Code: github.com/saezlab/SpaCEy

SpaCEy uses explainable graph neural networks to identify spatially organized tissue regions and molecular markers associated with outcomes such as patient survival and disease progression, without requiring predefined cell-type or tissue-region annotations.

Introducing mc-ASTRA, an open-source Python framework for studying how tissues vary and remodel across patients, perturbations, time points and other biological contexts. Manuscript: www.biorxiv.org/content/10.6... Package: saezlab.github.io/mc-astra/

This work was led by Ricardo Omar Ramirez Flores, jointly supervised with Julio Saez-Rodriguez. Klaus Kruger supported package development, and Francisca Gaspar Vieira and Loan Vulliard contributed to applications available in the documentation.

We also thank the MOFA-FLEX team for their support particularly (Ilia Kats, Florin Walter and Arber Qoku). We thank the European Research Council for their support through the Synergy Grant 101118531.

We have already used this general approach across heart failure, kidney disease, systemic lupus erythematosus and cross-organ fibrosis: www.nature.com/articles/s41... www.medrxiv.org/content/10.6... www.medrxiv.org/content/10.1... www.biorxiv.org/content/10.6... www.biorxiv.org/content/10.6...

The aim is to make these analyses easier to build, adapt and interpret, while retaining the flexibility needed for different biological questions and technologies.

With mc-ASTRA, we offer a flexible and accessible end-to-end framework for defining, constructing and interpreting multicellular tissue-state maps from single-cell and spatial data.

Powered by MOFA-FLEX (github.com/bioFAM/mofaf...), mc-ASTRA can combine molecular, compositional and spatial tissue descriptors with complementary sample-level information, such as time, clinical measurements, morphology or technical covariates, as well as biological prior knowledge.

Single-cell and spatial technologies now let us profile tissues in extraordinary molecular detail. Yet an abundance of data does not necessarily make it easier to distinguish technical from biological variation across samples, or to connect that variation to coordinated tissue-remodelling processes.

mc-ASTRA builds on our previous work on multicellular factor analysis (elifesciences.org/articles/93161), where we repurposed group factor analysis, pioneered by O. Stegle, W. Huber, J. Marioni, F. Buettner, R. Argelaguet, B. Velten and colleagues

21 Sept

How do we know whether a biomedical foundation model has learned something biologically meaningful - rather than simply learned to excel on a narrow benchmark? Our new Perspective by @juliosaezrod.bsky.social and @psl-schaefer.bsky.social with Nikolas Kalavros & Gustavo Stolovitzky discusses this 👇

doi.orgBenchmarking biomedical foundation models - Nature MethodsFoundation models are becoming increasingly popular across various biological domains, yet rigorous methods to comprehensively evaluate their full potential remain underdeveloped. This Perspective dis...

21 Sept

Perspective ➡️ doi.org/10.1038/s415...

embl.orgBenchmarking: the foundation of reliable AI in biomedicine | EMBLEstablishing shared evaluation standards to ensure AI models in biology are rigorous, reliable, and reproducible.

21 Sept

EBI news ➡️ www.embl.org/news/science...

Saez-Rodriguez Group reposted

EMBL-EBI

@ebi.embl.org

17 Sept

Closing the loop for reproducible science: 🧬 Data ➔ 🤖 Models ➔ 📏 Benchmarks As AI transforms biology, EMBL-EBI is building the infrastructure to ensure models are as rigorous as the data behind them. What’s your biggest challenge when validating AI models? www.embl.org/news/science...

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

Led by Francesco Carli, a research effort shared between saezlab & @opentargets.org, jointly supervised by @dugourdajf.bsky.social, d0choa.bsky.social & @juliosaezrod.bsky.social

With contributions from Polina Rusina, Lun Ai, @l-kuechenhoff.bsky.social, Paul Ka Po To, Ellen M. McDonagh, @slobentanzer.bsky.social, & Fabio Petroni 🧪

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

📦 Open source (Apache 2.0): github.com/biocypher/karenina 📖 Docs: biocypher.github.io/karenina

We look forward to the community's feedback. We hope Karenina offers a starting point to extend & build on existing benchmarks.

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

And an AI examiner reading each agent's code & every step it took could tell whether the biology went wrong or the software did: the difference between needing a better model & a better computing environment.

⚙️ Karenina provides a toolbox making benchmark creation, extension & evaluation friendlier: AI-assisted test drafting, a GUI & other utilities, open source & extensible.

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

💡 How do you judge an autonomous AI data analyst? We revisited BixBench (github.com/FUture-House...), an existing benchmark where AI agents analyse real biomedical datasets, & rescored its answers with Karenina.

Grading on a scale rather than pass-or-fail showed many "failed" analyses were nearly right. Changing the agent's runtime setup reduced one model's score (GLM).

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

💡 What if a user confidently tells a model its correct answer is wrong? Models often yielded on user pressure, a failure invisible when you test one question at a time. Tool usage didn't save them: many never rechecked, some yielded even after re-checking it.

Karenina also evaluates multi-agent systems: a second model set as guardrail, evaluated in the same run, flagged most reversals.

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

💡 Is a correct answer also a trustworthy one? We asked seven models, commercial & open weights, drug-discovery questions with & without the Open Targets Platform via an MCP tool. Tool usage improved correctness most where parametric knowledge was weakest.

Other dimensions told a richer story: some correct answers were unsupported by the evidence the model itself retrieved, tool access traded fabricated references for real papers cited on claims they don't support, & tool access multiplied token costs several-fold.

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

We built Karenina to evaluate models the way users experience them, across questions, conversations & autonomous agents.

Two bottlenecks stood in the way: 1) A single score can't tell you whether an answer is trustworthy enough to act on. 2) Experts who know what a good answer looks like can't easily build benchmarks themselves. Karenina tackles both. What did we find?

🧵 Introducing Karenina, an open-source framework to evaluate LLMs & agents: doi.org/10.64898/202...

In biomedicine, we score LLMs & agents far from how they are used: multiple choice nobody asks for, strict formats that test whether models follow instructions rather than what they know, & grading only the final verdict rather than the whole reasoning process.

20 Aug

Our tool MetaProViz is now out in Molecular Systems Biology: doi.org/10.1038/s443... Since our preprint we: - advanced our feature processing workflow - added metabolite-sets to MetSigDB - added metabolite-set visualization to aid functional analysis

Saez-Rodriguez Group@saezlab.bsky.social

We present our MetaProViz #Rpackage for #metabolomics analysis & prior knowledge integration to generate mechanistic hypotheses on how metabolic changes affect metabolite classes, pathways & environment interaction 🔗 www.biorxiv.org/content/10.1... 📦 saezlab.github.io/MetaProViz/ 🧵 Thread ⬇️

7 Aug

⬇️ Check out this new postdoctoral fellowship to be jointly hosted by Mathew Garnett's lab and our lab! lnkd.in/p/eZUYhWmz