Post

Great to see you doing rsearch something important. I didn't read closely, but it did look like you're using school-level data not individual student data. I suspect / wonder if the mismatch between experimental data and this has to do with this, a sort of ecological fallacy.

That is, thinking about it as a multilevel model with students nested in schools, you want level 1 (individuals) not level 2 (schools). Or at least, experiments are at level 1 typically, accounting for competing findings.

This is definitely a possibility (nonzero student effects, zero aggregate effects) and it's a part of the way I interpret the results in the paper. Notably though I also find no impacts on the lower or upper ends of the distribution either.

A lot of work on the impacts of AI on educational performance so far are experimental. They randomize student access to some specialized AI learning tool and often find positive results. But much of student AI use (esp where we expect negative learning effects) is not so controlled.

This strikes me as being a paper pretty uninformed about categorizing and measuring learning, then concludes with "AI neutral or good, overall", which makes me skeptical of the actual purpose of this exercise

I make use of the "summer dip" in ChatGPT usage that was observed in 2023 and 2024, where ChatGPT traffic fell way off when school let out. In areas with especially big dips (LOTS of student usage) did test scores drop?

Nope! Measuring aggregate test scores at the metro level in the US, I find no relationship between usage and test scores, across different grades in high school or subjects, or at different parts of the test score distribution.