For Better Science And Technology Policing, Don’t Forget The Archiving

I consider one of the things that distinguishes me in science policy circles (at least in the U.S.) is a facility to connect standard treatments of science and technology with relevant material from studies of same.  Disciplinary silos being what they are, it’s easy to overlook the overlap between these two.

This New Yorker piece summarizes a recent journal issue in advancing six ways to better police research.  Most of the suggestions involve, directly or indirectly, shifting current trends in publishing.  These shifts go beyond the moves beyond a  monoculture of publication outlets to include a healthy diversity of kinds of studies.  Meta-analysis, studies of failure, more replication work are all important to reassuring the public that while researchers can and do make mistakes, those mistakes can be found and corrected.

Besides changing incentives and norms in research and publishing (Retraction Watch and the Reproducibility Initiative shouldn’t be unique), such changes require new practices in archiving.  Fred Gibbs wrote about this earlier in the year, with a focus on science at risk – non-traditional sources of scientific output.  Also called ‘grey literature’ this would include blogs, research posted straight to the web, aggregations of science research and news, and other sources of research knowledge that aren’t journals.  In order to better capture efforts at reproducing other research, highlighting potential scientific misconduct, and finding research not available elsewhere, having more of this grey literature around becomes more important.

But it’s not that easy to do.  As with any proper archive, the challenges are found as much in making the collected information usable and useful than in gathering the material in the first place.  However, in addition to the questions and issues Gibbs outlines, there is the difficulty of capturing and maintaining an incredibly ephemeral media – the Internet.  When dollars committed to maintaining archives like, well, arXiv, run scarce, a lot of knowledge can disappear quickly.  The Internet writ large has this problem; governments have yet to effectively address it in a systematic fashion; even this blog hasn’t been able to prevent link rot from creeping into the earliest posts.

So, the tool that allows for wide dissemination and communication of research knowledge has within it the capacity to undermine it all.  Turns out we need the librarians and archivists probably more than we thought.