Klint Kanopka

Klint Kanopka

@klint.bsky.social

Assistant Professor of Applied Statistics at NYU ASH Psychometrician and sandwich lover Reluctant dog account klintkanopka.com

7.7K followers520 following128 posts

Pinned

Klint Kanopka
@klint.bsky.social

If you're trying to include LLM-generated item responses in psychometric work, you should check out my mixedsubjectsirt package that's now available on CRAN! Importantly, this is a statistical and LLM-agnostic approach with strong efficiency guarantees (a thread). klintkanopka.com/mixedsubject...

klintkanopka.comItem Response Theory Calibration with a Mixed Subjects DesignIntegrates large language model generated item responses into psychometric calibration studies through a mixed-subjects design for unidimensional two-parameter and one-parameter logistic item response...
Klint Kanopka
@klint.bsky.social

If you're trying to include LLM-generated item responses in psychometric work, you should check out my mixedsubjectsirt package that's now available on CRAN! Importantly, this is a statistical and LLM-agnostic approach with strong efficiency guarantees (a thread). klintkanopka.com/mixedsubject...

klintkanopka.comItem Response Theory Calibration with a Mixed Subjects DesignIntegrates large language model generated item responses into psychometric calibration studies through a mixed-subjects design for unidimensional two-parameter and one-parameter logistic item response...
View full thread
Klint Kanopka
@klint.bsky.social

If you're trying to include LLM-generated item responses in psychometric work, you should check out my mixedsubjectsirt package that's now available on CRAN! Importantly, this is a statistical and LLM-agnostic approach with strong efficiency guarantees (a thread). klintkanopka.com/mixedsubject...

klintkanopka.comItem Response Theory Calibration with a Mixed Subjects DesignIntegrates large language model generated item responses into psychometric calibration studies through a mixed-subjects design for unidimensional two-parameter and one-parameter logistic item response...
Klint Kanopka
@klint.bsky.social

Often LLMs can generate item responses that don't behave like human responses. Using a prediction-powered inference approach, we tune how much to rely on LLM data with the goal of minimizing downstream ability estimation error. If the LLM responses are trash, mixedsubjectsirt tells you to kick rocks

Klint Kanopka
@klint.bsky.social

We often don't consider it, but item parameter estimation error propagates into ability estimation error. Guarantees include unbiased parameter estimates and ability estimation error less than or equal to that of items calibrated on human subjects alone.

Klint Kanopka
@klint.bsky.social

Tomorrow at the PRIISM seminar: @crahal.com is going to talk to use about potential pitfalls and sources of error caused by psuedorandom number generation in social science research. Talk is at 11a eastern time on Zoom and open to anyone. RSVP here: steinhardt.nyu.edu/events/respo...

steinhardt.nyu.eduPRIISM | Responsible use of Pseudo-Random Number Generators in Applied Scientific ResearchJoin PRIISM and Dr. Charles Rahal to learn how to use pseudo-random number generators (PRNGs) responsibly in research.
Klint Kanopka
@klint.bsky.social

Wanted to hype up a software package I'm debuting today. If you do computer adaptive testing (CAT) research and are developing your own item selection/parameter update/exposure control algorithms, take a look at meow - a unified framework to make CAT simulation easier! klintkanopka.com/meow/

klintkanopka.comUnified Framework for Computer Adaptive Testing SimulationsProvides an extensible framework for conducting simulations to compare data generating processes, item selection algorithms, parameter update algorithms, and stopping rules in computer adaptive testin...
Klint Kanopka
@klint.bsky.social

Apparently some people like listening to me blather on about things! Thanks to @crahal.com for inviting me, this seminar has some amazing speakers lined up and brilliant attendees.

Charlie Rahal@crahal.com

💡🤖🔥The talk by @klint.bsky.social at metrics-and-models.github.io was truly brilliant. Important work; a clean/intuitive computational design & empirical application from the @itemrespwarehouse.bsky.social! *Next:* Juan Carlos Perdomo on "The relative value of prediction" (v. excited about this)💡🤖🔥

Klint Kanopka
@klint.bsky.social

Incredibly proud of the work @ruitings.bsky.social did, and it’s even more impressive that she just finished the first year of her master’s program!

Ruiting Shen@ruitings.bsky.social

I gave my first conference paper presentation at IMPS 2025 on process data analysis (credits to my amazing advisor @klint.bsky.social for all his support and advice) I also did my first data challenge and got a third prize (credits to my wonderful teammates Jessie and Na) #IMPS2025 #Psychometrics

Klint Kanopka reposted

Ruiting Shen
@ruitings.bsky.social

I gave my first conference paper presentation at IMPS 2025 on process data analysis (credits to my amazing advisor @klint.bsky.social for all his support and advice) I also did my first data challenge and got a third prize (credits to my wonderful teammates Jessie and Na) #IMPS2025 #Psychometrics

Klint Kanopka reposted

Wanjing Anya Ma
@anyawma.bsky.social

Thank you @psychonomicsociety.bsky.social for featuring our study! @jyeatman.bsky.social @klint.bsky.social

Psychonomic Society@psychonomicsociety.bsky.social

Assessing reading skills for an entire school’s worth of children is challenging and time-consuming – unless you’re using ROAR-CAT! 📚🦁Computerized Adaptive Testing (CAT) to assess reading abilities. Post by Anthony Cruz; #psynomBRM paper by @anyawma, @KlintKanopka and colleagues buff.ly/IuhWKaB

Klint Kanopka reposted

Charlie Rahal
@crahal.com

🔥Coming July 2: 'Metrics and Models'!🔥 A new seminar series on advanced modelling w/ impact across health/society. 🌍 Open to *ALL*! 🕑 Weds 2pm UK alternating weeks (online) 🔗 metrics-and-models.github.io 📬 metrics_and_models-subscribe@maillist.ox.ac.uk Brilliant speakers. *Please share*!

Klint Kanopka
@klint.bsky.social

Join us for our last (hybrid) PRIISM seminar of the academic year on 5/7! @austin-van-loon.bsky.social will present an experiment looking at how generative AI in admission essays may reshape the demographics of admitted students and how merit is evaluated. RSVP: steinhardt.nyu.edu/events/compe...

steinhardt.nyu.eduCompetitive Self-Presentation and GAI | PRIISMJoin PRIISM and Dr. Austin van Loon to learn about an ongoing online experiment investigating how generative AI in college admission essays may reshape the demographic makeup of admitted students and ...
Klint Kanopka
@klint.bsky.social

Join us at the intersection of causal inference and psychometrics as we host Josh Gilbert (@jbgilbert.bsky.social) at the PRIISM Seminar. Wednesday @ 10a: Estimating heterogeneous treatment effects with item-level data: Insights from Item Response Theory steinhardt.nyu.edu/events/estim...

steinhardt.nyu.eduPRIISM | Estimating heterogeneous treatment effects with item-level dataJoin PRIISM and Josh Gilbert to learn how to unmask hidden treatment effects within individual test items using Item Response Theory and to understand how to overcome the limitations of traditional si...
Klint Kanopka
@klint.bsky.social

Incredibly excited to have this finally come out! Model evaluation should be about comparisons, so we have a metric that puts comparisons in predictive performance on a common scale. I can’t make a thread about this better than @crahal.com, so I’ll let him take it away.

Charlie Rahal@crahal.com

1/5 📄✨ITS FINALLY HERE!📄✨. After _five_ years, our new model evaluation metric — The InterModel Vigorish (IMV) — is now published (@plosone.org). Paper: bit.ly/3RlqrQH Preprint: bit.ly/41LzGjm Code: github.com/intermodelvigorish ‘What’s the vig on this action?’ (bit.ly/4hPoZ3Z). ⬇️⬇️⬇️

Klint Kanopka
@klint.bsky.social

I've said this before, but the single greatest force holding back psychometric research is data availability. Researching innovative assessments requires response data, but those data are owned by those who own the assessments. This gives them carte blanche to gatekeep who can do this work. 1/2

Klint Kanopka
@klint.bsky.social

The elephant in the room, of course, is that in educational measurement, we use data collected from kids. The protections they need makes the logistics trickier. The Item Response Warehouse is one step toward democratizing psychometric research, but more must be done. datapages.github.io/irw/ 2/2

datapages.github.ioItem Response WarehouseOpen: The data are licensed for reformatting and public reuse. Their provenance is documented and code used to transform original data to meet the IRW data standard is available.Harmonized: The data a...

Klint Kanopka reposted

Andrew Ho
@andrewdeanho.bsky.social

Research with restricted-use data is more precise, powerful, and relevant. Faculty and staff will be working through the weekend to answer important research questions before we no longer can. This is not normal, necessary, or efficient.

AERA@aeraedresearch.bsky.social

AERA has just learned that all restricted-use NCES data licenses will be cancelled, possibly as early as March 20. We urgently request that all AERA members and others in the research community with restricted-use licenses take these two actions: www.aera.net/Research-Pol...

Klint Kanopka
@klint.bsky.social

The loss of the NAEP restricted-use process data is crushing. It was the most reasonable to acquire dataset that is useful for actual measurement work (as opposed to nonsense mining). The data hoarding practice of private ed and testing companies harms research and students. They must fill the gap.

AERA@aeraedresearch.bsky.social

AERA has just learned that all restricted-use NCES data licenses will be cancelled, possibly as early as March 20. We urgently request that all AERA members and others in the research community with restricted-use licenses take these two actions: www.aera.net/Research-Pol...

Klint Kanopka
@klint.bsky.social

I’ll add that I have a wonderful master’s student who has spent this entire academic year working to prepare to do interesting work with this data that was going to form the basis for her PhD applications. We’ll pivot and persevere as best we can, but she’s crushed right now.

Klint Kanopka
@klint.bsky.social

Can AI-powered textbots improve survey interviews? Join PRIISM and @soubhikbarari.bsky.social to examine how they can enhance response quality and scalability, while considering challenges in bias and respondent experience. If you're in NYC, join us in person! steinhardt.nyu.edu/events/ai-as...

steinhardt.nyu.eduPRIISM | AI-Assisted Conversational Interviewing: Effects on Data Quality and User ExperienceCan AI-powered textbots improve survey interviews? Join PRIISM and Soubhik Barari to examine how they can enhance response quality and scalability, while considering challenges in bias and respondent ...
Klint Kanopka
@klint.bsky.social

Today I talked about one of my favorite projects I've never published from. I did an internship at a Large Testing Company and before it ended, they sold the test I was working on to another company. No worries, great guy at LTC was helping negotiate data release for me. Then LTC laid that guy off.

Klint Kanopka
@klint.bsky.social

On Wed 2/12 at 10a ET, the NYU PRIISM Center is hosting Professor Alfonso J Martinez from Fordham University to talk about building adaptive assessments with continuous response formats. The talk is available online and open to the public. More info and RSVP here: steinhardt.nyu.edu/events/build...

steinhardt.nyu.eduBuilding Adaptive Assessments | PRIISMAdaptive assessments provide higher precision with fewer items. Join PRIISM and Alfonso J. Martinez to learn how to construct adaptive assessments with items that allow for continuous (as opposed to d...
Klint Kanopka
@klint.bsky.social

Come work with us at an amazing department in an amazing city with amazing colleagues and students. Happy to talk to anyone who is interested! Application review begins Feb 10. apply.interfolio.com/162021

apply.interfolio.com Apply - Interfolio {{$ctrl.$state.data.pageTitle}} - Apply - Interfolio
Daphna Harel@daphna.bsky.social

My amazing department at NYU is searching for a Clinical Assistant Professor - Applied Statistics for Social Science Research. This is a mostly teaching-based position, starting Sept 1 2025. (Application review begins Feb 10) I love working here so much! More info here: apply.interfolio.com/162021

Klint Kanopka
@klint.bsky.social

So proud of the hard work of @anyawma.bsky.social on this paper!

Wanjing Anya Ma@anyawma.bsky.social

My first PhD paper is out! I want to thank my advisors @jyeatman.bsky.social and Ben, and my collaborators (Amy, Adam, @klint.bsky.social , and Clem) and many people (including school partners) who supported this work! Nothing can be more rewarding to see ROAR-CAT helping classrooms every day! 🐈