Post

My radical suggestion for peer review: Move from nominal pre-publication review to explicit post-publication review. The original reason for peer review was that journal pages are a limited resource, so we need a filter before publication. That no longer makes sense with digital publishing. 1/

With the internet making publishing ~free, the limited resource is human attention, not publishing pages. It doesn’t make sense to use human reviewer time as a filter. And the filter is not exactly working well. 2/

Many papers are published on arxiv before being accepted at a conference. And rejected papers are just resubmitted elsewhere. What’s the point of the review process in this? We should just stop pretending that peer review is a meaningful pre-publication gate. 3/

So my suggestion is that we should review papers after they have been published – give the community some time to digest the paper before we formally evaluate it. Only allocate human reviewer time to papers that gain some attention first. E.g. only review papers once they start getting cited. 4/

What should happen to conferences like NeurIPS/ICML/ICLR? I think the attitude should be that anyone can present their work – subject to some very lightweight review and a strict one-per-registered-attendee limit. That way the focus is on the actual conference, not the associated review process. 6/

I was also thinking something similar, like, that the laws of free market could maybe work here too. That is, good papers get a lot of demand and bad ones are just neglected. And this is a strong signal already. But I was thinking this for preprints. 1/2

For conferences, even if people would ever agree that this is a good idea which they will not, it would be impossible to handle logistically; imagine 20k people showing up with their posters. It’s impossible both to host so many posters but also for people to not loose their shit with so much info.