Post

In a new preprint w/ @tianqizhan.bsky.social & @smfleming.bsky.social, we developed a new measure of metacognitive bias (over/under confidence in individual decisions)! doi.org/10.31234/osf...

But we have known for a while that this is a problem: mean confidence is confounded with response bias (c), sensitivity (d'), and metacognitive sensitivity (meta-d'), and we confirmed this in simulation (colors are two different responses)

Following a signal detection theoretic model of confidence ratings, we designed a new measure based on how much more evidence people need to make a response with high (vs low) confidence. It's easy to compute in an already standard workflow for metacognition research and has none of the confounds!

We then wanted to validate the measure in two ways. First, we found that under model misspecification, our measure is just as confounded as mean confidence. This is not ideal, but reflects similar findings that other measures of metacognitive performance are sensitive to modeling assumptions