Peer review affects the science we see
Going even deeper on pre-publication peer review
My friend and colleague Saloni Dattani has a brilliant essay on peer review in the latest issue of Works in Progress. She points out that pre-publication peer review is very new: it has only existed in roughly the last fifty years. Incidentally this is just the period in which scientific and productivity growth has seemed to decline.
She also points out how peer review systematically fails to catch a huge range of examples of scientific misconduct, fraud, and poor practice (see tweet below for a recent example). She shows how, despite this, it is a gigantic burden on researchers, and delays the publication of new research substantially.
researchers around the world spent a total of 100 million hours on reviewing papers in just 2020 alone. Around 10 percent of economics researchers spend at least 25 working days a year reviewing them. Today, a scientist who submits a study to Nature or PNAS, the Proceedings of the National Academy of Sciences, can expect to be published nine months later, on average. In the top economics journals, the process takes even longer – a staggering 34 months, or almost three years. And the length has been crawling upward each year.
And she shows how many scientific fields are, finally, starting to evolve around it, with things like pre-prints. Honestly, just read the whole thing, it’s an excellent article.
But I think that the problems that the peer review-publications-papers equilibrium introduces are even deeper than what she suggests in the post. Here is just one: it affects what we publish.
Academic careers depend, in the main, on publishing. Researchers who publish in better journals get better jobs, better grants, and most importantly of all, more prestige.
If someone scoops your idea, your follow-up might have dramatically lower impact, given how much less prestige is awarded for new results as opposed to replicating (or failing to replicate) previous results.
Combined with strict standards of publication, this has a range of predictable impacts on research.
Firstly, there is an incentive to make papers long. Researchers feel a need to cover off all possible criticisms of their idea, in order to get published. Partly, this is because their peer reviewers might in practice demand it in their own case. Equally, it is because they expect to see objections. This has been seen in a lot of fields – actually the paper I wanted to cite was in economics, but I can’t find it. Perhaps a commenter can.
Secondly, it favours a certain sort of novelty: novel empirically but not theoretically. Gatekeepers are selectively demanding of extra rigour for ideas that go against important paradigms in their field, driving a phenomenon where it takes literal deaths of big figures in a magisterium to see change. But striking new evidence that is anti-intuitive is favoured – leading to issues later when it is not replicated.
Most of all, it favours finished, complete nuggets of idea, even if they are not socially or scientifically important, over unfinished thoughts, hypotheses, and suggestions – since these are not publishable. It favours hard to criticise but valueless work over valuable but flawed work containing errors.
One might argue that these early ideas will find their way into the world regardless, and we can have a thriving ecosystem of unfinished, flawed, early, suggestive hypotheses alongside a vibrant world of rigorous, deep, apparently error-free academic papers. But this is not true. The need to keep your ideas close to your chest until you publish, so that you can get papers out of them, dooms this possibility.
If you simply throw your idea out into the world then you may receive all sorts of criticism – and the publishable elements of your research may be harvested by someone else entirely. Since there is no intellectual property for research ideas, releasing them is tantamount to losing your exclusive control of them.
This is quite different to the situation that obtained before peer review, when scrappy half-understood results in just a few pages were very often published.
But throwing ideas out into the aether is what allows collaborative, iterative, incremental tweaking. The sort of tinkering which powered the industrial revolution in the past, and powers rapid progress in speedrunning today.
So perhaps we should think even more radically than Saloni. Sadly, I haven’t got good answers on institutional reforms that are the next step. But in the spirit of doing peer review post publication, I am going to publish this post anyway.
P.S. There is a brilliant new issue of Works in Progress. Check it out here.