Post

... and that's where Registered Revisions comes in The Registered Revisions Meta Trial Pilot Report is OUT NOW!! osf.io/preprints/me... But new(ish) policy involves developing a new(ish) multi-headed trial design, workflows, infrastructure, piloting, mistakes, and a super team. And it's all here.

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

This report is a little different than what you might be used to. You won't see results or implications (due next year). This is a deep dive into the design and development of the policy, methods, ideas, etc. And most importantly, what didn't go to plan. Which was a lot. bsky.app/profile/whal...

Noah Haber@whaleactually.com

If you love any of the following, stay tuned for something I am stoked to share in the next few days: * Weird study design * Things super not going to plan * Workflow and infrastructure * Entirely exploratory preregistration * Fun stats/inference issues * Simulations

Registered Revisions is like a mini registered report for when reviewers/editors imply some new analysis or data collection be performed during standard peer review. It's what @dingdingpeng.the100.ci describes here, but lightly formalized and with developed workflows bsky.app/profile/ding...

Julia M. Rohrer@dingdingpeng.the100.ci

I know of at least one instance in psych in which the additional data collection was handled essentially as a registered report — that is, in principle acceptance for the whole package upfront, so that publication does not depend on the newly collected data turning out “right.”

To estimate its impact, we are doing something out of the box: a "meta trial" Rather than one big trial with a bunch of journals or a hopeful meta-analysis of independent trials, we do a bit of both. One centralized design and infrastructure, but with flexible independent journal-based trials.

1 repost4 likes

<record scratch> "You may be wondering how we got here". That's Part 1, and it starts with a curveball (no spoilers). Nearly everything in this project was developed by necessity (sometimes unexpected), from the policy itself to the meta trial design to the (non-obvious) outcomes selection.

Part 2: the Pilot, with a crew of kickass journal editors and co designers. Unlike standard pilots confirming or adjusting a well-developed trial, we needed to build this plane as we were flying it. The "pathfinder" pilot was more co-development than test implementation. Everyone pitched in a TON.

How about workflows and templates for integrating the Registered Revisions policy into standard editorial processes? Yeah we got that. You want infrastructure? You're covered. A whole webapp based system for participant management and tracking designed to work with normal editorial touchpoints.

Most importantly, what did the pilot journal editors (coauthors) think about the whole thing? What worked well, what was frustrating, and what went horribly wrong? We compiled our editorial team's thoughts so you can see for yourselves with (we think?) no sugar coating, good and bad alike.