Post

After a two year gap we have at long last resubmitted our paper on gender differences in metacogniton. I’m actually very glad we waited as it allowed us to utilize the newly developed brms package for meta-d, which was a massive upgrade addressing many previous issues. Updated preprint soon!

Will do an updated thread when the new preprint drops next week, but we added an entire 500 N replication sample and revised every analyses within a unified brms framework. Results have changed considerably.

Previously we reported a gender difference in both bias and efficiency, but the latter did not replicate. Crucially, Kevin O’Neil’s fantastic upgrades allowed us to estimate meta-delta, a new measure of bias that controls for d, c and meta-d. This showed a massive gender difference.

Overall the result is a very clear and replicable domain-graded gender difference in both local and global metacognitive bias. I can’t wait to share the full updated preprint and results with you next week!

This was the first time we really went all in on BRMS and I’m quite pleased we did. It really unified the modelling framework and allows us to have a highly consistent standard of evidence. Such a powerful framework. Sorry ANOVA, you are dead to me!

I learned a lot as a PI and an analyst on this one, and I think it’s a solid contribution. Sometimes, the right move can be to wait until things advance enough that the solution to a problem or limitation becomes clear! Everything on its own time.

While I’m Saturday musing, I really have to emphasize that this new fully hierarchical model solves so many issues with the previous hmeta-d framework. Estimating all type 1 and type 2 parameters solves quite a few issues with covariants and poorly estimated parameters.

Not only that but the brms toolkit makes estimating random and fixed metacognition effects much simpler, makes it far easier to test prior and posterior predictives, makes sensitivity analyses a breeze, and much more. I suspect a lot of results in the field will change.

Kevin knows I’m a big fan and he’s not on Bsky but just wanted to pour out some praise as it’s so nice when science progresses like that!