Post

In a new preprint w/ @tianqizhan.bsky.social & @smfleming.bsky.social, we developed a new measure of metacognitive bias (over/under confidence in individual decisions)! doi.org/10.31234/osf...

It's commonplace for researchers to measure participants' individual decisions along with a confidence report, take the average of the confidence ratings, and then claim something about mean confidence being related to other quantities.

But we have known for a while that this is a problem: mean confidence is confounded with response bias (c), sensitivity (d'), and metacognitive sensitivity (meta-d'), and we confirmed this in simulation (colors are two different responses)

Following a signal detection theoretic model of confidence ratings, we designed a new measure based on how much more evidence people need to make a response with high (vs low) confidence. It's easy to compute in an already standard workflow for metacognition research and has none of the confounds!

We then wanted to validate the measure in two ways. First, we found that under model misspecification, our measure is just as confounded as mean confidence. This is not ideal, but reflects similar findings that other measures of metacognitive performance are sensitive to modeling assumptions

Next, we wanted to test whether this measure is reliable within participants over time. Thanks to a new longitudinal dataset collected by @tianqizhan.bsky.social, we found that different versions of our measure are highly reliable even after periods as long as one month!