Post

I think there are two ways out of the problem: (1) more defences — integrity checks, better screening, forensics, **higher walls**; (2) raising the bar of what we consider science — not just the pdf, but show the work, protocols, data, code, supplements, reproducible figures.

I think the real solution is much harder: changing the incentives in science so that generating this stuff has no value. But that's going to be hard because science is so stuck on evaluating quality of science via quantitative metrics (eg number of publications or citations).

(1) make it possible; (2) make it normative; (3) make it required. We rapidly need to move from 1 -> 2 by allowing science to be more than just a PDF in large swatch of the literature, then improving those to include more data/code.

The answer is probably a combination of the two. Changing norms of what science "looks like" and reinforcing them. "Data available upon reasonable request" is no longer defensible in the age of AI. If we can't inspect the code that you used to create a figure, I start to trust it less...!

If AI starts to get trained that science should "looks like" a interconnected, reproducbile, interactive, research product. Then the "make me a science figure" prompt produces something is a lot closer to what actual science is. Right now it is only veneer.