Raphaël Merx

a *great* chart to teach confounding variables

This is some legit really impressive work!!

Introducing Global PIQA, a new multilingual benchmark for 100+ languages. This benchmark is the outcome of this year’s MRL shared task, in collaboration with 300+ researchers from 65 countries. This dataset evaluates physical commonsense reasoning in culturally relevant contexts.

Cool paper, at the intersection of grammar and LLM interpretability. I like that they use linguistic datasets for their experiments, then get results that can contribute to linguistics as a field too! (on structural priming vs L1/L2)

My favourite ICLR paper so far. Methodology, findings and their implications are all very cool. In particular Fig. 2 + this discussion point:

Incredible paper, finding that large companies can game the LMArena through statistical noise (via many model submissions), over-sampling of their models, and overfitting to Arena-style prompts (without real gains on model reasoning) The experiments they run to show this are pretty cool too!

Cool summary of issues with multilingual LLM eval, and potential solutions! If you're doubtful of all these non-reproducible evals on translated multiple choice questions, this paper is for you

GlotEval - a unified framework for multilingual eval of LLMs, on 7 different tasks, by @tiedeman.bsky.social @helsinki-nlp.bsky.social Just wish it supported eval of closed models (e.g. through LiteLLM?) github.com/MaLA-LM/Glot...

👋 Hey Bluesky! We’ve just touched down and we’re excited to be here 🌤️🐍 This is the official PyCon AU account, your go-to space for updates, announcements, and all things Python in Australia✨ Hit that follow button and stay tuned because we’ve got some awesome things coming your way! #PyConAU

😼SMOL DATA ALERT! 😼Anouncing SMOL, a professionally-translated dataset for 115 very low-resource languages! Paper: arxiv.org/pdf/2502.12301 Huggingface: huggingface.co/datasets/goo...

Been hearing a lot about recency bias lately. Must be pretty important

Our paper on generating bilingual example sentences with LLMs got best paper award @ ALTA in Canberra! arxiv.org/abs/2410.03182 We work with French / Indonesian / Tetun, find that annotators don't agree about what's a "good example", but that LLMs can align with a specific annotator.

Another example of why we need evals that take clinical risk into account when training NLP models for health slator.com/openais-whis...

What do it mean to be a “low resourced” language? I’ve seen definitions for less training data to low number of speakers. Great to see this important clarifying work at #EMNLP2024 from @hellinanigatu.bsky.social et al aclanthology.org/2024.emnlp-m...

this guy lives rent free in my hippocampus

Is productionisation (and move to gimmicks like CoT in o1-preview) at OpenAI and Anthropic a sign that scaling laws are slowing? And if so, where are we headed in LLMs? Slightly pretentious but enjoyable read: www.generalist.com/briefing/the...