Alexandra Olteanu

@shreyachowdhary.bsky.social

18 June 2025

We have to talk about rigor in AI work and what it should entail. The reality is that impoverished notions of rigor do not only lead to some one-off undesirable outcomes but can have a deeply formative impact on the scientific integrity and quality of both AI research and practice 1/

Alexandra Olteanu reposted

Shreya Chowdhary

I'm presenting this paper this afternoon in Malmo! Here's a peek at the poster, which previews the article's core arguments. Come find me to talk more about how we can reclaim democratic control over AI futures :) Read full paper here: doi.org/10.1609/aies...

Shreya Chowdhary@shreyachowdhary.bsky.social

New #AIES paper w/ @benzevgreen.bsky.social & @tisjune.bsky.social - "Deconstructing AI Hype: A Playbook for Resistance"! Read here: shreyachowdhary.net/files/decons... We argue that AI hype is a problem not because it misdescribes AI, but because it facilitates AI entrenchment.

I think a bigger issue than what benchmarks and other evaluations measure or not measure is making claims based on them that are simply not supported by what is actually being evaluated. Knowing what one can or cannot claim seems a much more common failure to me #AIES2016

Alexandra Olteanu reposted

Emma Harvey

@emmharv.bsky.social

I will be presenting "Data Annotation as Measurement" at #AIES2026 in Poster Session 3 (today @ 3:15PM CEST)! ojs.aaai.org/index.php/AI...

Emma Harvey@emmharv.bsky.social

✨New Work✨ forthcoming at #AIES2026: 1️⃣ "Data Annotation as Measurement" by me, @allisonkoe.bsky.social, and @kizilcec.bsky.social explores how data annotation can go wrong, and how measurement theory can help improve it. 🔗: arxiv.org/pdf/2608.07297

A quick reminder that we should not conflate *confidence* with *expertise* -- one helps you to notice whether an output is correct while the other does not #AIES2026

aicentre.dkP1 guest talk by Responsible AI researcher Alexandra Olteanu | Event | Pioneer Centre for Artificial Intelligence

I'm in Europe, and next week I'll be in Malmö (for #AIES until Weds) & Copenhagen speaking at the Pioneer Center for AI on Thurs www.aicentre.dk/events/20261... Thank you @gretawarren.bsky.social & @iaugenstein.bsky.social for hosting me! If you are around and want to chat/catch up, do reach out!

Alexandra Olteanu reposted

AI alignment is at the centre of the growing debates about AI systems and their failures. The impulses of AI alignment work are however *not* new. Making sure computational systems do what we want them to do has long been a core goal of computing research.

Differentiating between distinct components and the choices made to instantiate them also helps foreground many possible missteps practitioners risk making when implementing alignment techniques. This is work led by Matthew Bilik with Pınar Barlas, @rdbinns.bsky.social, @emtseng.bsky.social & me

papers.ssrn.comUnpacking AI AlignmentAI alignment has become a central focus of AI research and practice---cast as a response to the unprecedented scale, perceived autonomy, and generality of AI sy

A preliminary preprint is now online: papers.ssrn.com/sol3/papers.... A longer version is on the way, so feedback is welcome!

To illustrate this, we tease apart the constitutive components of AI alignment work (e.g. what is being aligned, along which system property, according to what criteria), alongside the definitional, operational, and epistemic choices made to instantiate those components.

To do so, we draw on both the system behavior specification literature from software engineering and on the measurement theory literature from the social science; two threads of literature that we also believe the AI community would benefit from engaging more with.

The very act of developing computational systems involves an intention about what the systems should or should not do (e.g. be helpful, not be harmful), which necessarily involves some notion of system behavior specification. We argue AI alignment should be viewed as a behavior specification device

scholarly-things.leaflet.pubOn (co-)authorship TL;DR — Authorship requires both earning it and being able to defend the work.

8 Aug

As we have discussions about e.g., “authorship quotas” for conferences and “authorship when using LLMs” it may also be worth thinking about how each of us approaches what authorship should entail. A few thoughts about how I think about this:

rigor-in-ai.leaflet.pubBack-to-basics: so what do evaluations doTL;DR — Developing useful evaluations requires clarity about what exactly one is trying to learn about a phenomenon of interest.

2 Aug

So what do AI evaluations do or should do? Wrote a brief reflection about prototypical questions evaluations often try to answer and why it is useful to have clarity about what exactly one is trying to learn about some phenomenon of interest. rigor-in-ai.leaflet.pub/3ms4ax4sojs2j

rigor-in-ai.leaflet.pubBack-to-basics: unobservable constructs and their ‘surplus meaning’TL;DR — Because unobservable constructs hold ‘surplus meaning,’ your metric is not your construct.

25 July

Wrote another brief post motivated by a reflection on how in AI research folks might not fully appreciate the care that working with unobservable constructs requires. TL;DR -- Because unobservable constructs hold ‘surplus meaning,’ your metric is not your construct.

rigor-in-ai.leaflet.pubBack-to-basics: on poor conceptualizations in AI workTL;DR — Poor conceptual foundations can severely undermine the credibility and reliability of knowledge claims. (And, no, your metric is not your construct.)

20 July

Wrote a brief reflection on poor conceptualizations in AI work and why they matter, something I believe is so startlingly neglected. TL;DR - Poor conceptual foundations can severely undermine the credibility and reliability of knowledge claims.

20 July

I'm experimenting with this format, but I'll see if I stick with it

@emilymbender.bsky.social

15 July

I'm thinking about writing brief reflections on a few topics (mainly from my own attempts to gain clarity about research practices I find concerning). I'm thinking about using medium, but I see many folks use substack. Are there other platforms? What's your reasoning for using a certain platform?

Alexandra Olteanu reposted

Emily M. Bender

7 July

I have a foolproof way to make sure that your papers contain absolutely no fake references . . . . . read the papers you cite.

Alexandra Olteanu reposted

minaremeli.bsky.social

@minaremeli.bsky.social

7 July

Come meet me at the last poster session at ICML if you are interested in chatting about LLM evaluation and pairwise comparisons! I will be presenting joint work with my advisor (Moritz Hardt). Thu Jul 9, 5:00 PM – 6:45 PM KST, Hall A (#4411) arxiv.org/abs/2606.09409

arxiv.orgCorrect Looks Better: Pairwise Comparisons Reveal Accuracy RankingsPairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge bi...

20 June

So often folks conflate outputs with outcomes; in most settings, the system outputs are different than the outcomes of interest.

30 May

Sometimes it feels like some folks are losing the plot about what the goal of publishing research actually is. The goal is certainly not meant to be the productions of papers, but rather the production and communication of science.

30 May

Those are not the same. Optimizing for publishing papers does not necessarily correlate with either more science or a clear understanding of what the science means. At least IMHO. <end of rant>

Yesterday was my last day at MSR. We recently learned that our roles were eliminated, and with them our little FATE Montreal team. I joined MSR a bit over 7.5 years ago while on active chemotherapy, and being at MSR has overlapped with so much change in my life.

It will take a minute for me to grieve and rest. I think it might also be the time for me to update my personal website, which has remained frozen in the summer of 2018. In the next period I am also hoping to gain some perspective, and have chats with folks.

If you want to chat about anything from research, impact, etc please do reach out! I will also soon likely start looking for short term sabbatical/visiting-like opportunities; so if you know of any such opportunities please do reach out as well.

... early work on evaluating text generation systems (e.g., autocomplete, smartreply) and understanding the ways in which such systems can fail (aka red teaming), early work on evaluating the quality of benchmarks and foregrounding critical construct validity issues, ...

youtu.beDehumanizing machines: Making sense of AI systems that seem humanYouTube video by Microsoft Research

... early work on foreseeing adverse impacts and performing impact assessments, and so much more. If you work on anything related to these topics you have likely read some of our work. This short video offers an overview of our more recent work on anthropomorphic AI systems: youtu.be/ZnLFDKX0KME?...

I also had the fortune to work with some of the most wonderful students and colleagues, and our little team has done so much rigorous, foundational work over the last 7+ years.

This includes some of the earliest work showcasing how a lack of conceptual clarity about the constructs under use when evaluating, building, or deploying AI systems hinders our ability to reliably operationalize or measure those constructs, ...

I joined as one of the first hires on the FATE Montreal team. When I found myself in the position of rebuilding the team a few years later, I poured my soul into doing so; this team has meant enormously to me.I am immensely proud of the work we've done over the years & of the impact our work has had

Alexandra Olteanu reposted

facctconference.orgACM FAccT 2026 Doctoral Colloquium Call for Applications

18 Feb

The deadline for the 2026 FAccT DC is next Tuesday, February 24! If you are a student working on topics relevant to the FAccT's scope, this is an opportunity to interact with a diverse set of peers and mentors! #facct2026 #facct26 #facct Details here: facctconference.org/2026/callfor...

Alexandra Olteanu@aolteanu.bsky.social

We are hoping to see applications from and bring together a diverse group of students across multiple disciplinary areas! If you are a graduate student and interested in FAccT’s scope, the Doctoral Colloquium is for you! #facct2026 #facct26

facctconference.orgACM FAccT 2026 Doctoral Colloquium Call for Applications

18 Feb

Alexandra Olteanu@aolteanu.bsky.social

facctconference.orgACM FAccT 2026 Doctoral Colloquium Call for Applications

21 Jan

ACM FAccT@facct.bsky.social

Calling all graduate students! The #FAccT2026 call is out for our Doctoral Colloquium. Join us for connections, support, mentorship, etc. Apply by February 24! facctconference.org/2026/callfor...

Alexandra Olteanu reposted

📣 It's again that time of the year 🤩 - our internship call for the FATE team @ MSR Montreal and our collaborators is now up! 🎉🎉 We are broadly looking for candidates interested in perceptions, evaluation, uses, and impacts of AI! #FAccT #FAccT2025 #CHI2025 #NeurIPS2025 #ACL2025

If you are interested in conceptual clarity, construct validity, benchmarks, and/or anthropomorphic AI behavior, design, or impacts you should consider applying! 🤩 If you are interested in consent and ownership frameworks for AI artifacts, I would really love to talk to you! 😍

Interested in any of these areas? Apply here: apply.careers.microsoft.com/careers/job/...

3 Dec 2025

@zey.bsky.social's paper and talk at ICWSM'14 (my first ICWSM!) shaped my thinking on issues around data/technology and the type of problems I still spend my time working on/thinking about. Can't think of a better keynote speaker for my first NeurIPS.

1 Dec 2025

On my way to @neuripsconf.bsky.social -- this is my first time at #NeurIPS and looking forward to seeing folks and presenting our paper on Rigor in AI. Do find me if you want to chat about rigor in AI, anthropomorphic AI, or evaluations #NeurIPS2025

Alexandra Olteanu reposted

Lucy Li

@lucy3.bsky.social

11 Nov 2025

It's the season for PhD apps!! 🥧 🦃 ☃️ ❄️ Apply to Wisconsin CS to research - Societal impact of AI - NLP ←→ CSS and cultural analytics - Computational sociolinguistics - Human-AI interaction - Culturally competent and inclusive NLP with me! lucy3.github.io/prospective-...

arxiv.orgRigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of RigorIn AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue th...

11 Nov 2025

Unexpected (amount of) snow day

Alexandra Olteanu reposted

Michael Ekstrand

@md.ekstrandom.net

7 Nov 2025

Replying to @md.ekstrandom.net

Our forthcoming NeurIPS position paper, led by @aolteanu.bsky.social, makes this argument (along with several related ones) in more depth. Rigorous AI/ML work should flow from explicit and rigorous premises, not just have a final evaluation that checks some rigor boxes. arxiv.org/abs/2506.14652

18 Oct 2025

I wish folks would use more precise terminology than "AI sycophancy." Not all validating behaviours/interactions are sycophantic. By definition, for them to be sycophantic there needs to be an underlying intention to e.g., gain advantage or favour. Intention is something AI systems do not have.

1 Oct 2025

Love this analogy

Olivia Guest · Ολίβια Γκεστ@olivia.science

People who coax chatbots into sensible answers are basically opening and closing the fridge until it contains something you wanna eat, yes, eventually you get hungrier & eat the stuff in there. But what changed was your cognition. The fridge stayed the same. You changed your mind about the contents.

29 Sept 2025

This was accepted to #NeurIPS 🎉🎊 TL;DR Impoverished notions of rigor can have a formative impact on AI work. We argue for a broader conception of what rigorous work should entail & go beyond methodological issues to include epistemic, normative, conceptual, reporting & interpretative considerations

Alexandra Olteanu@aolteanu.bsky.social

29 Sept 2025

As we prepare the camera-ready version of this paper, I am also reflecting on how to make this work handier and more useful: rigor cards to make the different facets of rigor easier to grasp? workshops to provide a forum for discussion and debates? something else that would be helpful to you?

Alexandra Olteanu reposted

Dr. Chinasa T. Okolo

@chinasa.bsky.social

12 Sept 2025

"Epistemic rigor, however, does not necessarily require specific epistemological commitments or choices but rather that those commitments and choices be made explicit." www.arxiv.org/abs/2506.14652

31 July 2025

Listening to a workshop panel at #acl2025 I am realizing that we are saying more or less the same things and having more or less the same conversations for so many years