Matt Makel

@aeraedresearch.bsky.social

17 Dec 2025

New Publication alert: Making a List Requires Checking it Twice: A Call for Empirical Evidence in Characteristics Lists [https://journals.sagepub.com/doi/10.1177/00169862251392934] #OpenAccess

Matt Makel reposted

Nick Brown

@steamtraen.eu

Current mood...

Matt Makel reposted

moin syed

@syeducation.bsky.social

22 Sept

New paper! Despite qualitative researchers claiming that generalizability is not a goal, their published work does indeed contain generalizable claims. We document this and provide recommendations for how all researchers can better calibrate their generalizability claims. doi.org/10.1177/2515...

Matt Makel reposted

AERA

9 Sept

Reminder! The Graduate Student Research-in-Progress Roundtable Series Call for Submissions is open. The deadline to submit to the call is October 15! Full eligibility and submission information here: www.aera.net/Education-Re...

Matt Makel reposted

Amanda Kay Montoya

@akmontoya.bsky.social

8 Sept

Curious about Zenodo as an OSF Project alternative? Here's a short video to get you started: File Uploads, Communities, and generating links for Anonymous Peer Review. youtu.be/jW84PxuJTMw

youtu.beGetting Started with Zenodo (transitioning from OSF Projects)YouTube video by AMANDA MONTOYA

Matt Makel reposted

Olúfẹ́mi O. Táíwò

@olufemiotaiwo.bsky.social

8 Sept

Replying to @olufemiotaiwo.bsky.social

if it took you 5 seconds of contemplation to think of an alternative you should take at least another 5-10 to at least consider the possibility that it at least crossed the minds of 1 or 2 of the people who dedicate their lives and research careers to dealing with the problems we post about on here

Matt Makel reposted

Cas Goos

@cas-goos.bsky.social

3 Sept

I am proud to announce that my article "Measurement reliability, construct validity, and transparent reporting in original and replication psychological research" has just been published in PLoS One. You can find the published article here: journals.plos.org/plosone/arti...

journals.plos.orgMeasurement reliability, construct validity, and transparent reporting in original and replication psychological researchPublished (replication) studies that use measures to assess psychological phenomena need to transparently report information on measurement procedures, validity, and reliability to allow verification ...

Matt Makel reposted

Randy Ellis

@randalljellis.bsky.social

2 Sept

Here's my conversation with @bethclarke.bsky.social, incoming Junior Professor of Psychological Metascience at the Leibniz-Institut für Psychologie. www.youtube.com/watch?v=2Hak... We discussed direct replications, how study limitations are ignored, and p-curving social and personality psychology.

youtube.comDirect replications, P-curves, and Acknowledging Limitations | Beth ClarkeYouTube video by Metascience Matters

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

That it took over a decade for this to come to light raises several questions. How can we improve the identification of such errors? How can we better platform those who observe a finding that does not sit well with them? How can we increase the rate and speed of correction? 21/

kclpure.kcl.ac.ukEstimating the prevalence of replication research is challenging, not least because even a thorough database search can be shallow.

Don’t forget to read the new paper kclpure.kcl.ac.uk/portal/en/pu... [non-paywall version] 22/22

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

I will be thinking about this for the rest of my career. I apologize to my co-authors and all readers of this work. I honestly thought we were searching the full text. I should have caught the error. 19/

Now that I know about the mistake, I am writing to the editors of the journals where I published in an effort to correct the record. My hope is that we can do so quickly. 20/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

In other words, exactly what we want to happen in scientific research (detection of errors to improve our understanding of the world) happened. …It just took much longer than I’d like. 17/

journals.sagepub.comSage Journals: Discover world-class researchSubscription and open access journals from Sage, the world's leading independent academic publisher.

As I wrote recently, “Long term behavior change upon acquisition of new information is useful. It's called learning.” I hope I never become someone who stops learning. I believe the field is stronger when we update our beliefs when we acquire new information. journals.sagepub.com/doi/10.1177/... 18/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

Perhaps ironically, these new results give me a deeply personal re-invigorated belief in the value of replication of research and the fallibility of well-meaning researchers and a flawed academic research publication system. 15/

Because of this fallibility, replication (particularly independent replication) is hugely valuable for helping us observe and understand the world around us as well as consume research findings. 16/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

journals.sagepub.comPutting the Self in Self-Correction: Findings From the Loss-of-Confidence Project - Julia M. Rohrer, Warren Tierney, Eric L. Uhlmann, Lisa M. DeBruine, Tom Heyman, Benedict Jones, Stefan C. Schmukle, ...Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion,...

Loss of confidence in research results is not rare. Public acknowledgement of loss of confidence is rare. journals.sagepub.com/doi/10.1177/... 13/

youtube.comselective attention testYouTube video by Daniel Simons

I am reminded of the famous invisible gorilla research in social psychology. The new results reveal that replications have been hidden in plain sight. We simply had to look in the right way to be aware that authors have been conducting and reporting replications. www.youtube.com/watch?v=vJG6... 14/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

Rakow deserves rich credit for helping reveal this error in framing. I thank him for this work. I owe him a sincere debt of gratitude for helping bring to light a substantive and meaningful inference change. I hope everyone interested in replication rate research reads this new paper. 11/

Science self-corrects when scientists take the effort to do so. It’s not automatic. And it’s not easy. Kudos and thank you to Dr. Rakow for doing it here. If I have a frustration with the new paper, it is that it didn’t come out about 14 years sooner. 12/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

Rakow: “My investigation suggests that many researchers are not shy about conducting and reporting replication. Plausibly, at least 20% of psychology articles include author-identified replication; and replication could be considered to be a mainstream practice in experimental psychology.” 9/

This new paper should meaningfully change beliefs and expectations about prior replication rates. How future work is done—and what we should expect from it—should be informed by these new results. 10/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

I would have bet money that many of the articles we coded included discussion of replication in text and not just in title/abstract/keywords. Of course, relying on the (obviously) faulty memory of individual researchers is an astoundingly terrible foundation for a scholarly field. 7/

Rakow’s new paper found that “around 90% of articles that discuss replication and over 80% of articles with author-identified replications are missed by searching databases for replicat* because databases hold few details per article”. 8/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

Database searches did NOT search “all text” as I believed. It did not have that capability. Instead, it searched only Titles, Abstracts, and Keywords of papers. 5/

In other words, the results we reported reflect the rate at which researchers were highlighting replications and not conducting replications. That is a meaningful difference. 6/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/

In 2012-2016, we used Web of Science to estimate a “replication rate” of how often researchers in various fields attempted to check (or conceptually build on) prior results. In general, we found that replications were quite rare (often < 1%). 3/

journals.sagepub.comSage Journals: Discover world-class researchSubscription and open access journals from Sage, the world's leading independent academic publisher.

a new paper by Tim Rakow of King’s College London reveals that prior work that relied on database searches to report replication rates underestimated the number of replications that have been conducted. journals.sagepub.com/doi/abs/10.1... 4/

I made a mistake. And because of that mistake, I no longer have confidence that results that I reported in several research papers are an accurate reflection of replication rates. 1/