Olmo van den Akker

Olmo van den Akker

@denolmo.bsky.social

Postdoc @ QUEST Center for Responsible Research & Tilburg University. Doing meta-research aimed at improving preregistration, secondary data analysis, and peer review.

285 followers241 following67 posts

Olmo van den Akker reposted

I am absolutely delighted to announce that the open access version of "In Referees We Trust? How Peer Review Became a Mark of Scientific Legitimacy" is now live! Want to know more about how scholars' lives came to be ruled by Reviewer 2? Download the book FREE at: direct.mit.edu/books/monogr...

Olmo van den Akker reposted

Olmo van den Akker reposted

Preregistration Authors see benefits but "concerns persist regarding limited perceived added value (e.g., unnecessary bureaucratization, increased workload), unintended side-effects (increased costs, discouragement of exploratory work), and reduced analytical flexibility." doi.org/10.1186/s410...

Big news at the META/e conference: metacheck, a promising tool for automatically extracting information from scientific manuscripts is beta-launched today. Has the potential to massively increase the efficiency of meta-science projects. www.scienceverse.org/metacheck/

scienceverse.orgCheck Research Outputs for Best PracticesA modular, extendable system for checking research outputs for best practices using text search, R code, and/or (optional) LLM queries.
Lisa DeBruine@debruine.bsky.social

@crist14n.bsky.social presented some of the capabilities of metacheck, which has its first beta release today! See an example report that demonstrates all of the current modules. www.scienceverse.org/metacheck/re...

Olmo van den Akker reposted

Olmo van den Akker reposted

📢 New from PYMS! We're launching online lunch meetings. A chance to present your work to other metascience ECRs Share your work in any form: finished projects, preliminary results, a trial-run conference presentation, or ideas you'd love feedback on Want to present? Sign up using the form!

forms.glePYMS - Lunch PresentationsBetween 20 and 24 April, PYMS will be hosting a series of online lunch meetings. During these sessions, you can present one of your completed research projects, an ongoing study, or early‑stage ideas ...

Olmo van den Akker reposted

Olmo van den Akker reposted

Now available on the OSF as part of a growing collection of preregistration resources, the new EEG & ERP Methods template guides researchers through every stage of ERP study planning. In our Q&A, two of its creators share how the template can help researchers at all stages:

cos.ioIntroducing the EEG and ERP Methods Template: Q&A with Gisela Govaart and Antonio SchettinoInterview with Gisela Govaart and Antonio Schettino, developers of the new ERP preregistration template on the Open Science Framework (OSF).

Olmo van den Akker reposted

“If fabricated studies make their way into the evidence base, they can mislead real scientists and ultimately slow progress for patients." Published in The BMJ, a new AI tool developed by @aidybarnett.bsky.social and colleagues has exposed the scale of fake studies flooding cancer research.

qut.edu.auNew tool exposes scale of fake research flooding cancer scienceA new machine learning tool has identified more than 250,000 cancer research papers that may have been produced by so-called “paper mills”.

Olmo van den Akker reposted

#OA articles are cited more, help citizen scientists learn more about the topic they've helped on, but there’s little strong evidence they have other long-lasting effects on research, or many economic and social benefits www.science.org/content/arti... @jeffreybrainard.bsky.social @science.org

science.orgIs ‘open science’ delivering benefits? Major study finds proof is sparseIt’s hard to measure social and economic impacts of making papers and data free, researchers say

Olmo van den Akker reposted

We are about a month away from releasing a complete refresh of the OSF user interface. The team has been working on this for a very long time, and we are very excited to be able to share it soon. A preview picture:

Many interesting tidbits in @simine.com's talk. A selection: - Journal prestige depends on factors like aims and scope, selectivity, and impact factor, but changes in these factors do not always lead to changes in journal prestige - journal prestige is sticky #PRC10

- "The replication crisis forced changes in transparency for the research itself, but not for the publication process" - We need to raise our expectations for journals? How? Nullius in verba (don't take their word for it!) #PRC10

- Journals should state what their aims and scope are from the outset and implement mechanisms to assess whether they achieve those aims. This could also be things like "we want to publish high risk research" - Meta-research is necessary to find out which journals deserve prestige #PRC10

eLife (talk by Nicola Adamson) uses a publish-review-curate method and uses common terms to assess manuscripts. For strength of evidence: exceptional, compelling, convincing, solid, incomplete, & inadequate For significance of findings: landmark, fundamental, important, valuable, & useful #PRC10

New peer review dataset incoming! Involves authors, topic area, editorial decision, author characteristics (institutional prestige, region, gender), BoRE evaluations, review characteristics (length, sentiment, z-score, reviewer gender). (Talk by Aaron Clauset) #PRC10

Christos Kotanidis checked differences in abstracts between submissions and published papers & assessed whether these differences indicated higher or lower research quality. Abstracts typically improved, especially in big five medical journals. Evidence for the effectiveness of peer review? #PRC10

Andrea Corvillon on distributed vs. panel peer review at the ALMA Observatory: Most experienced PIs no longer have the best ranks in a distributed review system, but why that is remains unclear. #PRC10

Interesting to see that the conference review process (and publishing norms) are do different in the field of computer science compared to other fields. How do these differences come about? Fundamental differences between fields or chance and inertia? #PRC10

Alexander Goldberg did it by a 7-point Likert scale for overall review quality but also by assessing 4 sub-categories: reviewers' understanding of the paper, whether important elements were covered, whether reviewers substantiated their comments, and the constructiveness of reviewer comments.

Ian Bulovic used OpenAI's GPT to assess selective outcome reporting. Findings: - Much outcome switching but decrease over time - Industry-sponsored trials most at risk - Assessing outcome switching may seem trivial but is even hard for human coders #PRC10

Ioannidis: Do we enough evidence for your proposed actions to improve peer review? Macleod: The evidence is thin, partly because many journals are hesitant to accommodate meta-research, like RCTs. #PRC10

John Ioannidis: "We need more RCTs" I agree, so here is an urgent call to the representatives of journals at #PRC10: Let's empirically test suggested improvements to peer review like open reports, open identities, structured review, results-free review, collaborative review, etc. Get in touch!

How do paper mills operate? I always thought they were in cahoots with illegitimate journals but apparently they target normal journals as well, with the editors of those journals playing no role. (Talk by Tim Kersjes from Springer Nature) Could open review reports solve this issue? #PRC10