Institute for Replication

1 Apr

🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science. Here’s what we found 👇

Already 115 registered participants within 24 hours. 10 slots left!

We are hosting Replication Games on Friday November 27th, 2026 at @unimarburg.bsky.social. Studies on conflict, political violence, and early childhood interventions for education and health will be reproduced! Register here: www.surveymonkey.ca/r/5Z3L7V8

Journal of Sports Sciences: Serious (new) data sharing policy and looking for a data editor. That's amazing and quite impressive given the norms in this field.

Looking for 125 researchers to participate in virtual forensic AI games! We will give you an AI report that documents irregularities in sports/exercise related articles. Also releasing our open source new I4R AI Audit Protocol. Details in this 🧵. Join us!

How good is the I4R AI Audit Protocol? That's what the games will measure: your human reports are the benchmark. We'll report false positives, misses, and what it gets right by the end of the year.

Join us (docs.google.com/forms/d/e/1F...) or have fun using our tool. We will release another AI reproducibility + robustness tool shortly. The current tool does not deal with robustness/sensitivity analysis.

Timeline: Complete 125 reports by Nov 13th. Review all reports by Nov 20th. Send all reports to original authors and journals by Nov 27th. Post all reports and authors’ responses by mid January 2027. Release a preprint and submit our meta-paper to a journal February 2027.

Why exercise/sports? It all started almost by accident with: x.com/I4Replicatio... Next is probably nutrition early 2027. Then another topic a few months later. Then another one...

Each participant will receive a completed AI report documenting reproducibility, potential data irregularities and coding errors. Task: Confirm the content of the AI report and write a short fully human report.

Our sample: 125 articles related to sports and exercise published in a few journals (Scandinavian J of Medicine & Science in Sports, Research Quarterly for Exercise & Sport, Psych of Sport & Exercise, J of Sports Science, J of Applied Sport Psych, Int J of Sport & Exercise Psych)

The tool is open source. You only need to connect your cli subscriptions in order to use it. However, for the purpose of the games you will receive the reports completed already.

github.comGitHub - I4Replication/AI-audit-protocol: A protocol for AI-assisted reproducibility auditing: specification, empirical agenda, and the skill that implements it (gate set G-2026-08-21)A protocol for AI-assisted reproducibility auditing: specification, empirical agenda, and the skill that implements it (gate set G-2026-08-21) - I4Replication/AI-audit-protocol

Tool: github.com/I4Replicatio... Paper: github.com/I4Replicatio...

Step 2: a coordinator merges their reports. Then critics from a different model evaluate every finding, for at least two rounds. A claim only stands if it survives, and it needs confirmation from two model families before it counts as settled.

Step 3: the output is a draft report plus a verification codebook that re-checks each claim in code. It reports what passed as carefully as what failed, never attributes intent, and never exposes personal data. That's what you'll get, and it's what you'll check during the games.

We built a new tool: the I4R AI Audit Protocol. It takes a published paper + its replication package and runs a full reproducibility & forensic audit, end to end. 🧵 How it works:

Step 1: five independent AI auditors each take a slice of the package: re-running the code and reproducing tables/figures, checking for duplicated or impossible values, coding & scoring errors, statistical-reporting anomalies (no data needed), and archive issues.

Register here: docs.google.com/forms/d/e/1F... Only graduate students and researchers holding a PhD. Contributors will be eligible for coauthorship of a meta-paper. Dates: Anytime between Nov 2nd and Nov 13th. Should be one day of work, but you have total flexibility.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

Based on the evidence we have examined so far, you would want much stronger validation before your favorite football team relied on these tests to identify its next star.

Is this game over? No, the replication games start in a few weeks! Stay tuned for more information.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

More importantly, perhaps, can psychological tests really help us find the next Messi?

Claims that cognitive or personality profiles can identify future elite athletes are potentially very consequential. Talent identification is a huge market, and professional teams in Brazil, Sweden, and elsewhere invest substantial resources in psychological and cognitive testing.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

One advantage we had in this investigation is that authors shared some data and sometimes code. We need to make responsible data and code sharing mandatory for everyone publishing in those journals.

Right now, only authors who share data can be rigorously investigated. And those authors might be the most transparent.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

Publishing a comment in economics is VERY hard, but at least it is theoretically possible in many journals. Here, the editors mentioned they would do an independent investigation, rejected our comment and told us not to contact the authors.

We contacted authors before the journals, so we did not obey this rule. The reward system to conduct forensic/replication investigation is beyond broken in those journals.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

We will make an announcement on Monday for AI games in which we will look at 125 articles in those fields. So we should know the answer by the end of the year.

Another question is: What are the incentives for doing this type of investigation? All journals we contacted so far (other than PNAS) rejected our comments simply because they do not publish comments.

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

Another group of authors also contacted us yesterday mentioning they will retract this article. See this thread for more details: bsky.app/profile/i4re...

Here are some thoughts on this investigation. Are the patterns documented outliers in the field of exercise, sport, and sport psychology? Would we find similar problems in representative sets of articles from these fields?

Two more retractions and a discussion of what we learned. 🧵 #CRAVEFiles

The first retraction is the PNAS article. PNAS investigated for months and finally retracted yesterday. In the notice, they allowed authors to respond. No mention of our work.

Notice writes: "A detailed account of the concerns, subsequent reanalysis, and reassessment can be found in Bonetti et al. 2026 [Institute for Replication, ideas.repec.org/p/zbw/i4rdps... (2026)].”" This is the authors' response, not our comment.

A researcher whose work we were examining asked our chair to investigate concerns about somebody else’s publications at the same university (different field). Our investigation was now generating further allegations from someone already under scrutiny. #CRAVEFiles

We will post this new report once we get a response from the authors.

Last thread tomorrow at 9AM ET. #CRAVEFiles Previous thread: bsky.app/profile/i4re...

Yet 14 rows match one PNAS team of 30 players. Two demographic rows differ; two other rows cannot be checked. Both papers cite the same ethics approval.

And there's more. BMC's "Corsi Blocks" results match a PNAS variable labeled "Forward Digit Span"—a different cognitive task. We also found numerous statistical inconsistencies.

Here's what we found. 14 of 18 rows in the BMC paper's Table 2 have IDENTICAL means and standard deviations to one team of 30 players in the PNAS dataset we recovered from deleted files in public Git history. Goals, assists, passes, cognitive tests, and more.

Here's the puzzle. The BMC paper reports 51 FEMALE players from 3 clubs. The PNAS paper describes 153 MALE players from 5 teams.

Those concerns deserve their own assessment, regardless of who raised them. But pointing us toward somebody else’s publications does not resolve the questions about the work we were already examining.

Remember the PNAS paper from our previous threads on elite soccer players and their cognitive abilities? We have now uncovered a striking connection to an earlier BMC Psychology paper. Same statistics. Different players?

We are paraphrasing rather than publishing the email: The sender asked our chair to independently check concerns about EIGHT other publications, request data and code where needed, and contact the journals.

A full zip folder with prepared emails for editors and statistical concerns for some of the articles was attached to the email.