“To preprint or not to preprint?” What’s the opportunity cost of early, non-peer-reviewed publicly available research?
NISO recently hosted a two day seminar in Washington DC entitled “Open Access: the Role and Impact of Preprint Servers,” representing stakeholders across #scholcomm discussing the perceived value, purpose, and implementation of preprint servers.
As a novice to preprints generally (and the nuances of genesis, value proposition, and current implementations specifically), I attended to “bone up” on all things preprints and because of a preview I received a week earlier. At the Charleston Conference Hyde Park debate, Oya Rieger of Itaka S+R and Kent Anderson of Caldera Publishing solutions debated whether preprints have improved the scholarly communication system.
Oya’s position on the benefits of fast, early communication of results, prior to peer-review, encapsulated the “pro” arguments I’m familiar with regard to preprints. Kent’s comments were new to me and when stripped of some of the agent provocateur language -- “preprints disrespect peer-review” -- raised many valuable points to evaluate when considering the role of preprints.
Now, don’t get me wrong, I ultimately come down on the side of PLOS with respect to our commitment to preprints (see an extensive overview on our blog regarding current preprint pilots); however, I do think the discussion at NISO merits deeper consideration (and for the full live tweet thread, see #NISOAccess19)
So! Do the benefits outweigh the concerns?
Various communities have demonstrated over decades that preprints are valuable, useful, and relatively “safe” in terms of exposing the broader public to research that is not peer-reviewed. As the opening keynote to the seminar, Kent Anderson made some interesting distinctions. First, he’s not anti-preprint (although his twitter feed sometimes conveys the contrary), he’s anti-preprint server (that is, the online platforms that disseminate preprints without taking responsibility for the quality of that content and potential downstream effects of making it discoverable). For the extended unpacking, view the #NISOAccess19 hashtag.
The argument is an interesting one: In a current information dissemination climate rife with alternative facts/misinformation and dominated by algorithms that encourage outrage (where anger = clicks = revenue), research findings -- peer reviewed or not -- that feed this ecosystem risk spreading like wildfire. And with closed media bubbles reinforcing belief systems held by groups, indifferent to facts, unverified, unvalidated scientific claims can be consciously manipulated to further political, social, economic, national security, or other agendas. At the very least they generate clicks, at the very worst they can instantiate beliefs that jeopardize, for example, public health.
Dog ageing: the perils of news reporting findings on preprint servers
And I certainly see this more and more. As a guilty-as-charged trashy news junkie, I stumble upon the infamous Daily Mail (DM) from time to time and they are continually publishing click bait articles. The most recent one I’ve seen is about dogs ageing quite differently than we have generally assumed. The research was featured in The Telegraph and The Times, which is probably how it found its way to the DM. The DM article certainly doesn’t link to the bioRxiv.org paper (here) and only mentions in passing that it’s not peer reviewed. The Telegraph mentions it as the closing line to the article.
When was the last time you got to the final sentence of an online article? Much less a click-bait one? (Note, it took me almost 10 minutes to get from the DM article to the bioRxiv article and none of the news outlets linked directly to it, just to bioRxiv.org. I had to search within the platform myself.)
Yes, non-peer-reviewed, dog age-ing research seems fairly harmless and mildly amusing. If it turns out, dogs don’t age like this, no one will die.
But once you get into research about vaccines, climate change, and politics, the cost/benefit calculus starts to change.
What happens when reputable outlets do it?
The famous case of the cell phones = brain cancer study still lingers. (I’m choosing to link to an article debunking it rather than the original paper in bioRxiv, which is still available.) Reputable outlets like Mother Jones and the New York Times published the findings and now most reliable sources have to preface their reminders that there’s no real evidence for this with extensive explanations of why people think there is. The American Cancer Society has to explain how cell phones work, and the National Cancer Institute, spends so much time explaining what radiofrequency radiation is that you might miss the section where they state that there’s no evidence to back these concerns.
What about when the far right does it?
The case of the far-right populist party, Vox, in Spain is an interesting and cautionary one. Social scientists across Spain have been raising the alarm over the Vox party’s use of research to justify their racist, xenophobic, and nationalist positions. Currently over 3,000 Spanish researchers have signed a petition to hold Vox accountable for their misrepresentation of scientific research.
They state in their petition:
“As such, VOX’s strategy amounts to nothing else than a facetious resort to using (allegedly scientific) data in order to pursue an ideological agenda impregnated by extreme nationalism, intolerance, racism and xenophobia. It is an agenda that misuses, derides, and ultimately mocks the work of thousands of social scientists, at the same time as it undermines the foundations of our polity through lies and fabrications. (emphasis mine)”
One such example of misrepresentation of social science research comes in their misrepresentation of a recent study done by the Centro de Investigaciones Sociológicos (CIS) on the Spanish public’s attitudes toward immigration. According to the Spanish newspaper, El Pais:
“Para sostener sus argumentos, el partido de Santiago Abascal se apoya en el CIS publicado este martes, según el cual "el 15,6% de los encuestados considera la inmigración como un problema para España", mientras que el cambio climático "solo interesa al 1,5% de los españoles". En realidad, cuando el CIS pregunta a los encuestados "cuál es el principal problema que existe actualmente en España", el 10.7% cita la inmigración entre los tres primeros y solo el 2,3% menciona los problemas medioambientales; lo que les sitúa en el 8º y en el 18º puesto, respectivamente, entre las preocupaciones de los españoles.“
“To support their arguments, Vox used data from a recent CIS survey claiming, “15.6% of respondents consider immigration a problem for Spain while only 1.5% care about climate change. In reality, CIS asked respondents, “What is the principal problem that exists right now for Spain,” and 10.7% mention immigration in their top 3 problems while only 2.3% mention environmental issues.”
The actual CIS study can be found here.
What does this have to do with preprints?
Well that’s the question. If misinformation coupled with systems primed to spread it is the primary concern of those, like Kent, who are worried about research that’s not peer-reviewed entering the mainstream, isn’t this already a problem? Peer-review isn’t the mechanism that stops malign actors for misrepresenting -- or entirely fabricating -- the “facts” they want to promote. (if InfoWars hasn’t taught us this, then nothing has).
When the Vox party manipulates research findings to further its messaging, it’s simply misrepresenting how respondents answered surveys. Peer-review isn’t preventing this kind of manipulation.
Indeed, at the NISO event, numerous speakers representing publishers, funders, preprints providers, and libraries focused on how the benefits outweigh those concerns. The fantastic visual (shown below), published in PLOS Biology in a 2017 paper on the benefits of preprints for Early Career Researchers (ECRs), illustrates what the various NISO speakers were identifying.
The NISO event further underscored these benefits. Whether it was the managing director of SSRN explaining their goal in creating SSRN -- “We wanted to recreate the hallway conversations” -- or the individual twitter feedback I received from ECRs during the seminar -- “I’ve had lots of great feedback on preprints that have substantially improved the manuscripts,” it’s clear that the communities building preprints see their value.
NISO speaker Thomas Narock, Assistant Professor in Integrative Data Analytics at Goucher College, noted that successful preprint servers come from communities that value them. They can’t be successful independent of that community. As Angela Cochran put it on Twitter: it’s not a “if you build it, they (the community) will come” situation. Communities must drive it.
And the ones that drive it, clearly value it. Jessica Polka, Executive Director at ASAPBio (currently a PLOS collaborator re: preprints), closed out the NISO event by underscoring these benefits in a presentation that speaks for itself. Slides 8-10 really speak to how ASAPBio has seen engagement with preprints evolve and what authors want out of them. It’s this kind of feedback that is really driving PLOS’s own approach to preprint and open peer-review collaborations.
The overall takeaway from the second day of seminars was thoughtfully synthesized by Jessica -- let’s not get lost in the navel-gazing as to what a preprint is. Rather, we should focus our efforts to generate a clear “vocabulary for the full suite of peer review and screening checks that can be applied to any version of an article in the publishing continuum.”
Much of the justified concern about the dissemination of research findings that aren’t peer-reviewed can be mitigated by using the checks and taxonomies appropriate to the field, clearing indicating the moderation strategies used by that community/server, and effective version monitoring so readers understand how the version they’re reading fits into a wider scheme community feedback.
With organizations like NISO, ASAPBio, and PLOS modeling these kinds of standards and behaviors, the benefits of preprints seem to far outweigh the costs (notwithstanding the serious question of the financial sustainability of preprints, which I haven’t addressed at all here but is a serious hurdle to their long term viability as a vehicle for dissemination -- but more on that another time).
