Martin Tutek

🚨🚨 New preprint 🚨🚨 Ever wonder whether verbalized CoTs correspond to the internal reasoning process of the model? We propose a novel parametric faithfulness approach, which erases information contained in CoT steps from the model parameters to assess CoT faithfulness. arxiv.org/abs/2502.14829

Models are sycophantic, and when they do challenge faulty or incorrect inputs, they're ambiguous and ineffective. In a new paper w/ a multilingual co-author team, we find that people may view LM responses that push back, yet still end up believing false content is true. arxiv.org/abs/2610.17703

Our (objectively awesome) program is now live! Join us on October 29th for an 🎢 interpretaganza🎢 in Budapest. Come and enjoy the great talks, insightful conversations and a surprising lack of slop (except for coffee*)☕️ *jk I am actually ok with conference coffee

Take a break from reading up on Navier-Stokes drama and add October 29th to your calendar, as we have a stellar lineup of speakers at this years' BlackboxNLP!

Due to the unprecedented amount of submissions to BlackboxNLP 2026, we will announce the decisions for Main, Special and ARR commitments on September 2 AoE. Thanks for your patience!

The @blackboxnlp.bsky.social reproducibility track is in an urgent need for reviewers! If you can review a paper (or two), these bite-sized papers deserve some attention, and reviewing them will leave you fulfilled! 🥰 Please reach out if you can help out! RTs appreciated 👐

Very happy that #IC2S22027 will be in Milan at Università Bocconi! I’m really looking forward to welcoming the computational social science community to my university and to Milan, and excited to be part of the team organizing it. See you in 2027! 🇮🇹✨

With a high volume of submissions this year, we're recruiting additional reviewers for BlackboxNLP 2026! 🗓 Review deadline: August 17 (AoE) 🗓 Review load: 2-4 papers 📝 Sign up: forms.gle/tha166UQZYqX...

For today's reading group, @veraneplenbroek.bsky.social presented "Old Habits Die Hard: How Conversational History Geometrically Traps LLMs" by Simhi et al. (2026) Paper: arxiv.org/abs/2603.03308 #NLProc

⏳ The BlackboxNLP 2026 Reproducibility Challenge deadline has been extended to July 24 (AoE) ⏳ If you've been working on a robustness check, ablation, or replication of recent NLP interpretability work, you have a bit more time to get your submission in.

⏳ One week left! The submission deadline for BlackboxNLP 2026 is July 17 (AoE). If you're working on analyzing or interpreting neural networks for NLP, now's the time to get your paper in. 📍 Co-located with #EMNLP2026 in Budapest 🔗 blackboxnlp.github.io

🏆 Announcing the NDIF Best Paper Award for the BlackboxNLP 2026 Reproducibility Challenge! @ndif-team.bsky.social Reproduce an interp. finding with nnsight + NDIF, open-source it, and push it further. Winners will receive a $500 prize, and will be invited to present at the workshop!

At ICML '26, TakeLab members are presenting three papers. Make sure to stop by if interested in language model interpretability, coding agent evaluations, or pluralistic alignment. Details below 👇🧵 @icmlconf.bsky.social #ICML2026 🇰🇷

📣 #EACL2027 updates: the Call for Papers is live, and our keynote speakers are confirmed! Main conference: 9–14 March 2027. Special theme: The Human in Language. 🧵👇 2027.eacl.org/calls/papers/

We don’t always know what problems are hard for LLMs. So devs evaluate on tasks HUMANS find hard or on broad benchmarks. What if we could instead anticipate which scenarios a model will fail on—all without evaluating specific input examples? 🧵NEW PAPER by @jenniferlumeng.bsky.social

✨ it's coming ✨ NEMI 2026 will be lit. It will also be the new BU interp supergroup's debut ball. Come meet us!

With the large influx of submissions and a faster pace of research, reproducibility is more important than ever. With this reproducibility challenge, we want to put the focus on best practices wrt. baselines🧱, ablations🌈, eval🔎 and generalizability🗺️ of interpretability!

📣 Announcing the BlackboxNLP 2026 Reproducibility Challenge! A new track dedicated to rigorous robustness checks of NLP interpretability work - stress-testing baselines, ablations, generalizability, and evaluation.

Bingo! Do I win a prize now?

Ever used a top-ranked LLM that just... felt wrong for you? You’re not alone. Instead of leaderboards, many of us turn to "vibe-testing" - manually comparing models to our own needs. But can we turn these feelings into a structured evaluation? New paper: "From Feelings to Metrics" 🧵

We are delighted to welcome @marlutz.bsky.social to our lab over the next few months! 🎉 She'll work on the representation of different demographic groups in LLMs. #NLProc

FYI #ACL2026 has an unusual registration system this year, and probably a lot of people who want to attend will not be able to. Spots are limited to 3.5k people, and only presenting authors can register during the first phase. Then, *if* there are spots left, others can try to register.

How can generative AI better support human creativity, without limiting it? If you have thoughts, we invite submissions to our ICML workshop on Generative AI, Creativity, and Human-AI Co-Creation 📍 July 2026, Seoul 📄 Submit by: April 24 (AOE) 🔗 Submission link: openreview.net/group?id=ICM...

❗The full paper submission deadline for COLM is ~14 hours from now (11:59pm AOE)! Please submit your final PDFs on the same page where you uploaded your abstracts. And please use the provided LaTeX templates; do not handwrite your manuscript like this llama is! Good luck!

Interested in pursuing a PhD in NLP/cog-sci? Studying language learning in LMs from the perspective of human language acquisition? Few more days to apply!!

A piece co-authored by an old friend (Divya Saini, a psychiatrist at Massachusetts General Hospital) www.nytimes.com/2026/03/29/o...

Check out works on sequence repetition 🔁 and evaluating synthetic data 🧮 from our lab in Rabat! @eaclmeeting.bsky.social #EACL2026

Thinking of applying for an #MSCA Postdoctoral Fellowship in 2026? I’m open to supervising at Bocconi! Feel free to reach out. By submitting an expression of interest to Bocconi, selected applicants will receive full proposal support. 🗓️ Deadline: April 15 👉 www.unibocconi.it/en/horizon-e...

Excited to present this work together with @dippedrusk.com at #EACL. Join us in the poster session 1 (11:30-13:00) 🔥

Excited to share that @milanlp.bsky.social will be presenting 5 new papers at #EACL2026 and workshops in Rabat 🇲🇦!

Argh this sucks. apparently they lost their funding guarantee

You are #EACL2026? Check out some great work from the CSS Department @gesis.org, our data Science Methods team, with great collaborators!

I’m seeing close to zero reaction/conversation about this on here. This is huge news for open research on language models, especially in the US.

“It’s toddler AI misinformation at an industrial scale. It’s very risky for the developing brain.” Children’s media experts say AI-generated “slop" has infiltrated the internet, preying on young children and their unsuspecting caregivers.

We'll have a reproducibility track at this years' Blackbox workshop! Details are still within a slightly opaque box. We want to see if cleaning solutions that make opaque boxes 📦 transparent 🍱 work on different boxes 🎁📮🧰🥡, and with different 🧽solution-to-water🧼ratios!

NSF Update through March 13, 2026 1/2

#MemoryModay #NLProc 'Make Natural Language Processing About People Again' by @dirkhovy.bsky.social (2018) uncovers how AI models portray different religions and emotions. #AIEthics

Sharing “Neural Thickets”. We find: In large models, the neighborhood around pretrained weights can become dense with task-improving solutions. In this regime, post-training can be easy; even random guessing works Paper: arxiv.org/abs/2603.12228 Web: thickets.mit.edu 1/

📢 PhD position in Developmental Language Modelling (PLZ RT) What can human language acquisition teach us about training language models? Join us as a PhD! mpi.nl/career-education/vacancies/vacancy/fully-funded-4-year-phd-position-developmental-language @carorowland.bsky.social @mpi-nl.bsky.social