Noah Haber

cos.ioWhat is in the Registered Revisions Meta Trial Development and Pilot Report?COS Principal Research Scientist Noah Haber shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-sour...

Up on the COS.io blog now: "What is in the Registered Revisions Meta Trial Development Report" www.cos.io/blog/what-is... Yeah this is a post about a post about a report. But it's an unusual report worth a read, since it focuses on all the lessons learned, dead ends, straight up errors, etc.

cos.ioToward Collaborative Reproducible ManuscriptsCOS Principal Research Scientist Noah Haber shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-sour...

17 Sept

Got a preview of the upcoming Quarto 2 collaborative editor a few months ago, and it's genuinely very very cool and huge step toward practical collaborative code-driven manuscript workflows. May become my default for solo-ish projects or where collaborators are at least a little code savvy.

Richard Meyer-Eppler @r-meyereppler.bsky.social

The problem is well-recognized. The upcoming Quarto 2.0 will introduce a collaborative editor on the web as an attempted solution. Not sure how viable this will be, but certainly an interesting development! opensource.posit.co/blog/2026-04...

opensource.posit.coWhat's next: Quarto 2We've started working on [`quarto-dev/q2`](https://github.com/quarto-dev/q2/), a full rewrite of Quarto in Rust.

17 Sept

Can't beat Google Docs and similar for writing collaboration for larger teams and/or less code-oriented coauthors, so the placeholder-style workflow lives on for now. But having more good options and closing that gap is a huge win.

14 Sept

I have a paper that is about the fundamental and immutable severity of our causal inferential powers and our tendency to fool ourselves about them and it has a hidden Missy Elliott lyric and title that's a backronym from the Blondie comics. Sometimes, you just need the levity to get through.

Robert Talisse@roberttalisse.bsky.social

I once had a paper rejected at a journal because one reviewer said my writing was "too witty" for "the gravity of the essay's topic." Go figure.

14 Sept

Why yes I am 40 how did you know

14 Sept

For the curious: doi.org/10.1016/j.an... or free version on arXiv arxiv.org/abs/2004.04251 Comes complete with an R package to auto reveal assumptions hidden in your DAG to make you hate it/yourself

Noah Haber reposted

Center for Open Science

@cos.io

9 Sept

⬇️ The Registered Revisions Meta Trial Pilot Report documents the development & pilot of a new approach to peer review that asks authors to precommit to substantive revisions before conducting new analyses. It also details the multi-journal trial design, workflows, infrastructure, & lessons learned.

Noah Haber@whaleactually.com

... and that's where Registered Revisions comes in The Registered Revisions Meta Trial Pilot Report is OUT NOW!! osf.io/preprints/me... But new(ish) policy involves developing a new(ish) multi-headed trial design, workflows, infrastructure, piloting, mistakes, and a super team. And it's all here.

Noah Haber reposted

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

1 Sept

If you love any of the following, stay tuned for something I am stoked to share in the next few days: * Weird study design * Things super not going to plan * Workflow and infrastructure * Entirely exploratory preregistration * Fun stats/inference issues * Simulations

This kind of sausage-making what-went-wrong and what-else-did-we-try process stuff is crucial to the sciencing. But it's rarely disclosed, and even rarer to be the main focus of an entire design and development paper. (please someone give me a better analogy than sausages i hate it)

IT'S OUT NOW!!!! osf.io/preprints/me... bsky.app/profile/did:...

osf.ioOSF

Noah Haber@whaleactually.com

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

And there's some bonuses in there. You want simulations???? Heck yeah you do. We needed some way to understand the whole Registered Revisions and policy process, plus bonuses like pre-writing our analysis code based on simulated data. How about a live demo webapp? Yup.

So check it out! We think it might have genuine potential to really change the way we do both policy impact evaluation and peer review. I am STOKED to show eveyrone the process and not just the result (main results coming next year). We want your thoughts! Reply, email, takedown thread, whatever.

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

How about workflows and templates for integrating the Registered Revisions policy into standard editorial processes? Yeah we got that. You want infrastructure? You're covered. A whole webapp based system for participant management and tracking designed to work with normal editorial touchpoints.

Most importantly, what did the pilot journal editors (coauthors) think about the whole thing? What worked well, what was frustrating, and what went horribly wrong? We compiled our editorial team's thoughts so you can see for yourselves with (we think?) no sugar coating, good and bad alike.

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

<record scratch> "You may be wondering how we got here". That's Part 1, and it starts with a curveball (no spoilers). Nearly everything in this project was developed by necessity (sometimes unexpected), from the policy itself to the meta trial design to the (non-obvious) outcomes selection.

Part 2: the Pilot, with a crew of kickass journal editors and co designers. Unlike standard pilots confirming or adjusting a well-developed trial, we needed to build this plane as we were flying it. The "pathfinder" pilot was more co-development than test implementation. Everyone pitched in a TON.

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

Registered Revisions is like a mini registered report for when reviewers/editors imply some new analysis or data collection be performed during standard peer review. It's what @dingdingpeng.the100.ci describes here, but lightly formalized and with developed workflows bsky.app/profile/ding...

Julia M. Rohrer@dingdingpeng.the100.ci

I know of at least one instance in psych in which the additional data collection was handled essentially as a registered report — that is, in principle acceptance for the whole package upfront, so that publication does not depend on the newly collected data turning out “right.”

To estimate its impact, we are doing something out of the box: a "meta trial" Rather than one big trial with a bunch of journals or a hopeful meta-analysis of independent trials, we do a bit of both. One centralized design and infrastructure, but with flexible independent journal-based trials.

Jeremy Freese@jeremyfreese.bsky.social

Lurking within the new Ariely scandal is the incentive problem of editors giving R&Rs that ask for new analyses (that need to turn out a specific way) or even in psych whole new experiments (ditto).

This report is a little different than what you might be used to. You won't see results or implications (due next year). This is a deep dive into the design and development of the policy, methods, ideas, etc. And most importantly, what didn't go to plan. Which was a lot. bsky.app/profile/whal...

Noah Haber@whaleactually.com

<refresh> <refresh> uuuuuugh come onnnnnn <refresh> * human mods do your thing as long as you need, i appreciate you

Still under moderation, but a teaser: Personal favorite bit of the report is getting to talk about what went wrong in a way that I don't often see in an academic-ish paper. Mistakes made, ideas scrapped, and even a particularly bad code bug that I personally am responsible for (and fixed).

Noah Haber reposted

Paul Whaley

@dangerwhale.bsky.social

1 Sept

Spoiler alert: Noah tried to do research with editors. I feel like someone should've warned him.

Noah Haber@whaleactually.com

13 Aug

Coming out of bluesky lurking as a former northeasterner/ north carolinian climber, now colorado-ite, to note: Mt. Washington is the true avatar of the northeast: mean, small, and ready to mess you up in a HURRY if you disrespect it. Also fastest officially recorded windspeed anywhere on earth.

Northeasterners understand that the shortest distance between two points is a straight line. The trails were built before switchbacks had even been invented.

13 Aug

I repeat: Mt. Washington, at a mere 6.3k ft / 1.9k meters, has the record for the Fastest. Recorded. Windspeed. On Earth. Ever. You can mess yourself up anywhere in the mountains if you want to, but Mt. Washington has a penchant for messing you up just because it's grumpy that day.

The west coast may have a lock on altitude training, but Mt Washington produces world class Marco Polo champions every winter.

13 Aug

We used to have a rule that you never under any circumstances let Mt Washington know you are there to climb. You are there to take the gear for a walk. You might be exploring ideas for future climbing. But never ever are you there to actually climb rock or ice or anything up there. No sir.

There are two kinds of people: those who see a serene landscape in this photo, and those who know. Source: sectionhiker.com/mt-washingto...

Noah Haber reposted

Center for Open Science

@cos.io

⚙️ New on the COS blog: Principal Research Scientist Noah Haber (@whaleactually.com) shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-source R package. www.cos.io/blog/toward-...

cos.ioToward Collaborative Reproducible ManuscriptsThe first active phase of COS's Benchmarking LLM Agents on Scientific Tasks project has produced ReplicatorBench—a benchmark for evaluating LLM agents on research replication in social & behavioral sc...

For most scientific papers, coathors collaborate in the manuscript doc and someone manually enters analysis results/tables/figures into it. In this case, reproducibility effectively stops at the data and code. What this blog post presupposes is, what if it didn't? www.cos.io/blog/toward-...

In a reproducible manuscript, the data, analysis code and narrative text are all knit together, making a full pipeline from data/code to final manuscript automated and reproducible. This idea has been around forever, but these workflows usually make collaborative writing difficult.

This blog post shows how we made a *collaborative* reproducible manuscript workflow for SCORE papers It was not only possible, but by far the most *practical* option. The post discusses workflows for reproducible manuscripts, a demo R package for making them, and worked examples. Enjoy!

7 May

When I started working in global health a little over 10 years ago, infectious disease (particularly policy) was said to be a declining field to be almost entirely replaced with chronic disease. That ... did not happen.

Had a small part of this that I am pretty proud of: Full push-button reproducible manuscripts from data to manuscript. Nearly every stat, figure, number, table, etc is pulled directly from analysis code and data and "knit" with manuscript text written in Google Docs. Blog post coming soon ...

nature.comReliable research in the social and behavioural and sciencesSweeping new investigations probe the replication, robustness and reproducibility of results across the behavioural and social sciences.

Today, the SCORE program releases its primary results! 865 researchers examined research credibility across the social and behavioral sciences, publishing three papers in Nature + five preprints. 📑 Explore the papers: www.nature.com/colle... ℹ️ Read more about SCORE: www.cos.io/score

Profoundly weird to see these charts in the science mainstream, but if you want to remake them it's all in the code side note: the color palettes for the above linked reproducibility and replicability papers were based on samples from Weezer album cover art. you're welcome.

Nature@nature.com

Nature research paper: Investigating the reproducibility of the social and behavioural sciences go.nature.com/3OhvkfP

go.nature.comInvestigating the reproducibility of the social and behavioural sciences - NatureA study of reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences finds higher reproducibility among more recent papers and papers from journals that require data sharing.

13 Apr

~100 downloads of the reproducible manuscript packages now, and so far no contacts about errors or not being able to get the package to run etc. Looks like things are working, or at least not obviously horribly broken. Neat.

update: internal muffled scream

2 Apr

Been seeing a few people getting hit with the Nature paywall, so fun fact: Using the analyst code/data packages above, you can computationally reproduce these papers in their entirety, free and open on osf.io. (they're also free and open on MetaArXiv osf.io/preprints/me...)

Reproducible manuscripts have been around since code existed, but they are still pretty rare and mostly written in scripting langauges (text and everything) like RMarkdown. For SCORE, the text is written collaboratively via Google Docs, enabling modern collaborative writing with a large team.

I hope y'all tear into it and find any lurking issues, questions, ideas for your own work, etc. Gotta say it's a bit terrifying to have the entire pipeline so fully exposed and accessible, particularly one as complicated and high profile as SCORE. But more importanty, I hope it's useful.

Reproducible manuscripts are useful for a few reasons: One of the best reasons is just practicality. The thing most of us do where we have to manually update every stat/figure etc in a manuscript every time something changes in the code or data is super error-prone and time consuming.

Then there is rigor and accountability. Anyone can trace every stat and figure to *exactly* how it was estimated. If there's an error or a way you think things should have been done instead, it's super findable. As a bonus, that incentivizes code readability, since people might use it.