Effectiveness of Journal Ranking Schemes as a Tool for Locating Information

Abstract

BackgroundThe rise of electronic publishing [1], preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review [2]. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information [1]. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research.Methodology/Principal FindingsHere, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.Conclusions/SignificanceOur model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

Analysis of bibliometric indicators to determine citation bias

“…Data have been obtained for research papers published in the 1995-2004 period. This 10-year period was chosen because papers published during this period already have a sufficient number of citations for reliable statistical analysis, and because the rate of accruing new citations is relatively small compared to already accumulated citations, indicating the steady state of citation distribution (Stringer et al, 2008). Only regular research papers published in journals were analysed.…”

High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information

“…So, this does not seem to be a practical method of generating forecasts from very narrow early citation windows. Stringer et al show that in the long run the cumulative citations to articles published in a given year in a given journal that are cited at least once converge to a lognormal distribution [24], [25]. They term the distribution when no further citations are accumulating the steady state.…”

“…I use three functional forms to test the sensitivity to different specifications, though many more are obviously possible. The models are loosely based on the results of Stringer et al [24]. The first regression model assumes that:…”

“…I fit the Poisson model to the data to see how well this works in comparison to the models that assume that the dependent variable is continuous. Again, loosely based on Stringer et al [24], for all those articles with at least one citation by year t, I assume that the log of the mean of the distribution of the dependent variable can be modeled using:…”

How Atypical Combinations of Scientific Ideas Are Related to Impact: The General Case and the Case of the Field of Geography

“…The switching algorithm preserves the total citation counts to and from each paper, and the distribution of these citations counts forward and backward in time to ensure that a paper (or journal) with n citations in the observed network will have n citations in the randomized network. For both the observed and the randomized paper-to-paper citation networks, we aggregated counts of paper pairs into their respective journal pairs to focus on domain-level combinations (Itzkovitz et al, 2003;Stringer et al, 2008Stringer et al, , 2010. In the data, there were over 122 million potential journal pairs created by the 15,613 journals indexed in the WOS.…”