How Science Moved Beyond Peer Review During The Pandemic

When papers from China began flooding the websites bioRxiv and medRxiv in the first months of 2020, it was a strange and notable change. Founded as places where scientists could post drafts of research papers before those papers went through a traditional peer review, these sites had never really advertised much in China or gotten many submissions from scientists in that country before, said Richard Sever, the co-founder of bioRxiv and medRxiv. The sudden shift turned out to be a preview of the pandemic to come. “We got a wave of submissions from China and then a wave of submissions from Italy. And I remember being with a colleague, looking at submission numbers, and the chart was so eerily familiar,” Sever said. “It looks just like the progression of pandemic caseloads.”

“Preprint,” or “prepress,” servers have been around for decades, but during the COVID-19 pandemic, they took on a new notoriety and level of importance. Two of these sites alone, bioRxiv and medRxiv, hosted 25 percent of all COVID-19-related scientific research published during the pandemic’s first 10 months — more than 10,000 papers. In contrast, only 78 preprints were uploaded to bioRxiv during the entire 2015-16 Zika epidemic. (The medRxiv site didn’t exist yet.) COVID-19 proved to scientists that preprint servers are crucial resources in a crisis. At the same time, though, experts said the pandemic also made the shortcomings of scientific publishing clear. Preprints share a lot of the same limitations as peer-reviewed research, especially when it comes to how the media and the public use research once it’s out in the world for all to see.

Websites that post scientific research before it has been peer-reviewed and published in a scientific journal have been active since at least 1991, when the classic arXiv site, originally used mainly by physicists, went live. Even before that, scientists have shared drafts and notes among themselves through personal correspondence ever since “science” as a field became a thing. Those lines of communication exist parallel to the traditional process of peer review, which puts gatekeepers between research and publication. First, a paper has to be accepted by the editors of a scientific journal. Then, it goes to a (usually anonymous) panel of other scientists for critique. Their notes will lead to edits, which usually lead to more experiments, and then (hopefully) eventual publication. 

But the peer-review process does not happen quickly. The editors are sorting through thousands of dense submissions, and the reviewers are volunteers reading in their free time. On average, it takes about six months for a life-sciences research paper to go from acceptance to publication. “During the outbreak we maintained the same peer-review and editing process, just on an accelerated schedule … but I don’t think that this is a sustainable model,” said Dr. Eric Rubin, editor in chief of the New England Journal of Medicine. 

He and other scientists said the issue of speed was particularly important for clinical research during the pandemic because doctors needed the most up-to-date information to make life-saving decisions for critically ill patients. For example, the large trial of the steroid dexamethasone — which first demonstrated that this cheap, widely available drug could reduce the likelihood of death in COVID-19 patients — was originally posted as a preprint. It would show up, traditionally published, in the New England Journal of Medicine eight months later, but the preprint made it possible to get that knowledge into doctors’ hands faster. “I had one MD who contacted me, and he said, ‘You know, there are probably people who are alive today who would have been dead if not for preprints,’” Sever said.

Successes like that have contributed to making preprints more widely accepted among biological and medical scientists. It also helps that research in preprints isn’t a lot different statistically from how those same papers eventually end up being published in a peer-reviewed scientific journal. A 2020 study of pre-COVID-19 bioscience that compared preprint papers and their later, peer-reviewed versions, for example, found that the biggest differences were in details like how clearly the title reflected the conclusions or how easy it was to find relevant information in the article.

But that should be understood as a critique of the peer-review process rather than a glowing endorsement of preprint paper accuracy, said Alice Fleerackers, a graduate student and researcher at the Scholarly Communications Lab, a joint project of the Simon Fraser University and University of Ottawa. “There’s really a perception that peer review is a trustworthy quality-control mechanism,” Fleerackers said. But research hasn’t been able to support that idea. All the jokes and paranoia about COVID-19 being spread by 5G cellphone technology, for instance, trace their origin to a paper published in — and later retracted by — a scientific journal that claims to peer-review its content.

“We’ve seen with lots of the misinformation and many of the retractions that came out during COVID … ‘high-quality’ — and I’m using air quotes here — research in peer-reviewed publications can be just as flawed as research in preprint,” Fleerackers said. 

The real trouble with preprints — which is, funnily enough, also the real trouble with peer-reviewed research — is how those studies are promoted and written about on social media and by the press, experts told me. 

“Prior to the pandemic, I had fewer concerns about poor-quality science being preprinted and then widely disseminated,” said Maia Majumder, a computational epidemiologist at Harvard Medical School. “But now, everyday people are reading them too — and the media is covering them at a rate that far outpaces pre-2020. This means that when bad science is preprinted and happens to be sensational enough to get traction, it can quickly shape the discourse around a given topic.”

One of the top-tweeted and top-reported-on preprints in the first 10 months of the pandemic was a study that tried to estimate the share of the population in Santa Clara County, California, who had already been infected with COVID-19. The results, which found evidence for a higher rate of infection in the general population than anyone had previously calculated, was used to support the idea that COVID-19 would cause few deaths and that no lockdowns or distancing measures would be necessary. It was heavily critiqued and estimates had to be revised downward before BuzzFeed News revealed the research had been funded in part by an airline magnate who was critical of COVID-19-related social restrictions.

It’s unclear how much of the study’s coverage in the media was actually reporting the findings as opposed to the criticism and controversy it caused, but Fleerackers’s research suggests digital media outlets aren’t routinely providing readers with the context they need to understand the limitations of preprint studies. In a study published in January, Fleerackers found that, about half the time, digital publications made no mention that a COVID-19 preprint study being discussed even was a preprint. 

Reporting on preprints without explaining their limitations risks misleading the public the same way that reporting on peer-reviewed research as inherently accurate does, Fleerackers said. Though this reality was certainly known before the pandemic, the importance of preprint servers over the past year and a half has really driven the lesson home. Servers like bioRxiv and medRxiv have added prominent disclaimers to their pages. University press offices have begun to institute new rules that control the way preprint articles are promoted to the media. There’s even a new journal specifically dedicated to reviewing hyped COVID-19 preprints — and debunking them, if necessary. 

But the pandemic has also made clear how much the information in a preprint slips out of scientists’ control the moment it is posted online. “I hope they’re developing an awareness,” said Fleerackers, “that if you put something in public it is fair game for journalists, and for the public … and for conspiracy theorists.”