Assessing open source software as a scholarly contribution

Abstract

Introduction Academic computer science has an odd relationship with software: Publishing papers about software is considered a distinctly stronger contribution than publishing the software. The historical reasons for this paradox no longer apply, but their legacy remains. This limits researchers who see the open-source software movement as an opportunity to make a scholarly contribution. Expanded definitions of scholarship acknowledge both application and discovery as important components. 1 One obstacle remains: evaluation. To raise software to the status of a first-class contribution, we propose "best practices" for the evaluation of the scholarly contribution of open-source software. Typically, scholars who develop software do not include it as a primary contribution for performance reviews. Instead, they write articles about the software and present the articles as contributions. This conflation of articles and software serves neither medium well. An article describes an original intellectual contribution consisting of an idea, the argument for its importance and correctness, and supporting data. In contrast, software is more often an implementation of prior ideas in a usable form. It bridges the often considerable gap between an idea and the practical application of that idea. The original idea and its implementation represent distinct kinds of contribution. The critical gap is the perceived incomparability of these two contributions. Lacking a concise description adapted to the traditional practices of performance review committees, software is difficult to evaluate as a scholarly contribution and is often relegated to second-class status. We propose a framework for common assessment based on widely accepted definitions of scholarship. Within this general framework, we consider the material and procedures that a performance review committee uses to evaluate a publication. We then describe how software can be summarized in a compatible form of bibliographic citation and supplementary material.

Incentives and integration in scientific software production

“…One common suggestion is to reward academics for contributions other than journal articles, including data sets [35] and, more rarely, software [17]. Our work suggests that the specific form of non-journal contributions that would be acknowledged would be important in encouraging collaboration.…”

“…In short reputation in academia is about scientific contribution, whereas reputation in open source is about software work. Scientific contribution, for better or worse, is most often measured through publications and citations, not through other artifacts [17]. This has two implications.…”

On the evaluation of research software: the CDUR procedure

“…Software publications, as it happens already for research data papers, are becoming progressively installed in some scientific areas. However, in our opinion, there is still not a real RS publication procedure of comparable status as the one achieved for research articles, that is, well established and widely adopted by the scientific community (see for example 28). Thus, we cannot rely on the concept of a research software paper to fix some features towards a precise RS definition.…”

Towards computational reproducibility: researcher perspectives on the use and sharing of software

“…Seemingly small alterations, even for well described and openly available software tools, can lead to significant effects on analytical outputs (McCarthy et al, 2014), a problem exacerbated by the fact that researchers often have minimal formal training in software development practices (Hannay et al, 2009;Joppa et al, 2013;Prabhu et al, 2011). The iterative and collaborative nature of software development also means that it does not fit easily within existing academic incentive structures (Hafer & Kirkpatrick, 2009;Howison & Herbsleb, 2011;Howison & Herbsleb, 2013), which makes it difficult to create incentives to follow best practice.…”