Post

One solution: normalize desk-rejections. After a fast first assessment, papers are either desk-rejected, or go to full review. The current conf peer-review system has desk rejects as the exception, as it relied on self-selection prior to submission and small numbers. Neither of these hold anymore.

Andrew Cullen@adrcullen.bsky.social

Peer review relies upon reciprocity - I'll check your work, knowing that you'd do the same for me. Your investment in writing deserves my investment in reading. However AI generated theory papers are really testing my patience, and my peer reviewed faith. #ICLR

2 reposts55 likes

Desk rejects are typical in Bio, and increasingly common in Comp Bio IMO, despite the obvious benefit of accelerating slop processing, they are a net negative as: 1) they greatly reward established players; 2) they narrow the scope of work being published to "trendy" topics. I hope CS can keep away

Hard disagree. Theory venues are already super conservative. This time STOC is even single blind. They’ll just desk reject anything that isn’t super standard or authored by big name authors. I am sorry but I don’t trust STOC reviewers with desk rejections.

Im really confused about ICLR this year. I recommended 1/3 of my batch for desk rejection (which I think was generous), but no action was taken (there was no response) so the reviewers are reviewing these papers anyway. I think it makes sense to have an actual desk rejection period before review.

Interesting! Based on what did you recommend desk rejections? Clearly AI-generated papers? Clearly AI-generated and clearly bad? Just clearly bad? For comparison, in my AC batch I didn't see anything that I felt comfortable to desk reject. But perhaps I did not spend enough effort on that.

One paper was complete gibberish. One was readable, like parseable English, but said nothing at all (reviews have started coming in and seem to agree). One was incorrect format and so unclear if it was within the length requirement. And one was missing required sections.

Fully agree that there's a need for greater proportions of desk rejections, or even 2-phase systems where reviews only get passed onto the full set of reviewers if the first phase reviewers mark it as acceptable. Anything to take the voluntary workload of reviewing more seriously.

All my reviews for ICLR declared LLM generation, which was within the rules. However the theory paper was just theory, no validation - and while superficially it looked fine, it was impossible to validate properly without a huge time investment, an investment that the authors seemingly skipped.

Computer security has moved to this model, but IMO for frivolous instead of useful reasons. Ie instead of desk-rejecting slopful papers, they are rejecting papers which have, eg, minor formatting issues, or which dont have valid orcids…

It does sting to get a desk rejection after one week. What else does? Waiting 4+ months for the same rejection, with a 2-line review (or, worse, an entirely AI-generated one). If you're not going to engage with the paper, make it quick.