On the Predictability of Future Impact in Science

Abstract

Correctly assessing a scientist's past research impact and potential for future impact is key in recruitment decisions and other evaluation processes. While a candidate's future impact is the main concern for these decisions, most measures only quantify the impact of previous work. Recently, it has been argued that linear regression models are capable of predicting a scientist's future impact. By applying that future impact model to 762 careers drawn from three disciplines: physics, biology, and mathematics, we identify a number of subtle, but critical, flaws in current models. Specifically, cumulative non-decreasing measures like the h-index contain intrinsic autocorrelation, resulting in significant overestimation of their “predictive power”. Moreover, the predictive power of these models depend heavily upon scientists' career age, producing least accurate estimates for young researchers. Our results place in doubt the suitability of such models, and indicate further investigation is required before they can be used in recruiting decisions.

Understanding the Scientific Enterprise: Citation Analysis, Data and Modeling

“…This is considerably a more complex problem. For example, a recent attempt to predict the future evolution of the h-index of authors, based on linear regression models applied to their past performance [81], turned out to be largely unsuccessful [82], highlighting that cumulative impact measures (such as the h-index) are not suitable bases for prediction approaches.…”

Actions Speak Much Louder Than Words

“…For example, it has been reported that the future h-index 6 of an investigator is moderately predictable on the basis of the current h-index, number of articles, publications in prestigious journals, and diversity of the journals in which the articles are published 7 ; however, the h-index would not be appropriate to inform funding decisions because this metric has several flaws, including its intrinsic autocorrelation and the fact that its predictive power is heavily dependent on career age and is least accurate for young researchers. 8 The annual number of citations at the time of prediction has been claimed to be the best predictor of future citations. 9 The number of publications and citations in the previous 5 years has been found to correlate with the citation impact of a grant.…”

Scientific Production and Productivity in Curriculum Vitae Characterisation: Simple and Nested H Indices that Support Cross-Disciplinary Comparisons

“…Note that I will disregard individual characteristics that change over time, such as the researcher's age or position (Cainelli et al, 2012(Cainelli et al, , 2015Morichika & Shibayama, 2015), and individual characteristics that are fixed in time, such as gender or race (Abatemarco & Dell'Anno, 2013;Hopkins et al, 2013;Sotudeh & Khoshian, 2014). In this paper, my interest is in indices for, rather than determinants of, scientific activity (Penner et al, 2013). Moreover, I will omit papers that use indices to compare countries (e.g., Sangwal, 2013), institutions (e.g., Abramo et al, 2013c), or journals (e.g., Ko & Park, 2013;Tsai, 2014).…”