Deep impact: unintended consequences of journal rank

Abstract

Most researchers acknowledge an intrinsic hierarchy in the scholarly journals (“journal rank”) that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.

Imagining the ‘open’ university: Sharing scholarship to improve research and education

“…If the majority of OERs are produced by prestigious U.S. universities, it represents 126 another form of exclusion and reinforces the problem of Western perspectives (and the English language) 127 dominating educational content [44,50,52]. Resource-rich universities in Canada, the U.S., and Europe 128 should look for ways to support, raise visibility, and increase the use of OERs from other countries with 129 diverse global perspectives to facilitate a "true knowledge exchange" [44]. An example of an OER project 130 from Africa is the Science Education Exchange for Sustainable Development (SeeSD; www.seesd.org), based 131 in Senegal, which is designing open resources to improve access to education and STEM participation.…”

“Excellence R Us”: university research and the fetishisation of excellence

“…This is particularly true at higher prestige journals (Resnik et al, 2015;Siler et al, 2015;Belluz, 2016). If we add to this list of (potentially) "false positives" studies that cannot be replicated, the number of papers that meet one measure of "excellence" (that is, passing peer review, often at "top" journals) while failing others (that is, being accurate and reproducible, and/ or non-fraudulent) rises considerably (Dean, 1989;Burman et al, 2010;Lehrer, 2010;Bem, 2011;Goldacre, 2011;Yong, 2012b;Rehman, 2013;Resnik and Dinse, 2013;Hill and Pitt, 2014;Chang and Li, 2015;Open Science Collaboration, 2015). It is the very focus on "excellence", however, that creates this situation: the desire to demonstrate the rhetorical quality of "excellence" encourages researchers to submit fraudulent, erroneous, and irreproducible papers, at the same time as it works to prevent the publication of reproduction studies that can identify such work.…”

Citations and science

“…Although it has never been designed as such, the annual reports of the SCI are used as tool to assign funding to study-groups or institutions [5]. It also is used in scientific institutions to give a score to the research quality of researchers or research departments.…”

“…Clarivate Analytics is a very closed organisation; their criteria for assigning groupings, or accepting journals into the Index are not always very clear, and even the way the IF is calculated is not fully transparent [5]. In this case, however, we would be very interested in seeing a response to the paper of Minguet et al, because they discuss an issue that is very important for research level assessment and funding at large, and not only for practice related research in the field of pharmacy.…”