On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper

Glänzel, Wolfgang

doi:10.1016/j.joi.2010.01.005

Cited by 32 publications

(18 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Using the Leiden Rankings, the Center for Science and Technology Studies (CWTS) multiplied the product of the number of publications P with the “old” crown indicator CPP / FCSm to obtain as a result what was called the “brute force indicator.” In the new set of Leiden indicators, analogously, a “total normalized citation score” was proposed (Van Raan Eck, van Leeuwen, Visser, & Waltman, 2010, p. 291). However, all these indicators are based on the parametric assumption (of the central limit theorem) that one is allowed to compute with the mean as a summary statistic given a sufficiently large number of observations (e.g., Glänzel, 2010).…”

Section: Discussionmentioning

confidence: 99%

Integrated impact indicators compared with impact factors: An alternative research design with policy implications

Leydesdorff¹,

Bornmann²

2011

J. Am. Soc. Inf. Sci.

127

View full text Add to dashboard Cite

In bibliometrics, the association of "impact" with centraltendency statistics is mistaken. Impacts add up, and citation curves therefore should be integrated instead of averaged. For example, the journals MIS Quarterly and Journal of the American Society for Information Science and Technology differ by a factor of 2 in terms of their respective impact factors (IF), but the journal with the lower IF has the higher impact. Using percentile ranks (e.g., top-1%, top-10%, etc.), an Integrated Impact Indicator (I3) can be based on integration of the citation curves, but after normalization of the citation curves to the same scale. The results across document sets can be compared as percentages of the total impact of a reference set. Total number of citations, however, should not be used instead because the shape of the citation curves is then not appreciated. I3 can be applied to any document set and any citation window.The results of the integration (summation) are fully decomposable in terms of journals or institutional units such as nations, universities, and so on because percentile ranks are determined at the paper level. In this study, we first compare I3 with IFs for the journals in two Institute for Scientific Information subject categories ("Information Science & Library Science" and "Multidisciplinary Sciences"). The library and information science set is additionally decomposed in terms of nations. Policy implications of this possible paradigm shift in citation impact analysis are specified.

show abstract

Section: Discussionmentioning

confidence: 99%

Integrated impact indicators compared with impact factors: An alternative research design with policy implications

Leydesdorff¹,

Bornmann²

2011

J. Am. Soc. Inf. Sci.

127

View full text Add to dashboard Cite

show abstract

“…The methodological dimension should handle how to evaluate what, providing an appropriate account of reliability and robustness (Glänzel, 2010;Glänzel & Moed, 2013), and uncertainty.…”

Section: Methodsmentioning

confidence: 99%

A Framework for the Assessment of Research and Its Impacts

Daraio

2017

Journal of Data and Information Science

View full text Add to dashboard Cite

This paper proposes a holistic framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions, including in an original way, data as a main dimension, together with theory and methodology. Each dimension of the framework is further characterized by three main building blocks: education, research, and innovation (theory); efficiency, effectiveness, and impact (methodology); and availability, interoperability, and “unit-free” property (data). The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as “quality.” Three additional quality attributes are identified as implementation factors (tailorability, transparency, and openness) and three “enabling” conditions (convergence, mixed methods, and knowledge infrastructures) complete the framework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. It can act as a common denominator for different analytical levels and relevant aspects and is able to embrace many different and heterogeneous streams of literature. Directions for future research are provided.

show abstract

“…• excluding newly emerged researchers due to their relatively low citations and paper numbers (Costas and Bordons, 2007); • ignoring the ups and downs in researchers' scientific work, in that their cited papers may increase in number despite not having a newly published paper (Costas and Bordons, 2007); • remaining in relatively small level of publications due to the inability to exceed the number of published papers; • ignoring the number of co-authored authors (Glänzel, 2010) and the lifetime of an individual scientist's scientific work (Egghe, 2006); • difficulty in collecting all needed data (Costas and Bordons, 2007); • not decreasing in spite of no increase in the number of citations to the paper in years (Kelly and Jennions, 2006); and • insensitivity to lowly cited papers and not prioritizing highly cited papers.…”

Section: Literature Reviewmentioning

confidence: 99%

Multiple h-index: a new scientometric indicator

Yaminfirooz

Gholinia

2015

The Electronic Library

View full text Add to dashboard Cite

Purpose -This paper aims to evaluate some of the known scientific indexes by using virtual data and proposes a new index, named multiple h-index (mh-index), for removing the limits of these variants. Design/methodology/approach -Citation report for 40 researchers in Babol, Iran, was extracted from the Web of Science and entered in a checklist together with their scientific lifetimes and published ages of their papers. Some statistical analyses, especially exploratory factor analysis (EFA) and structural correlations, were done in SPSS 19. Findings -EFA revealed three factors with eigenvalues greater than 1 and explained variance of over 96 per cent in the studied indexes, including the mh-index. Factors 1, 2 and 3 explained 44.38, 28.19 and 23.48 of the variance in the correlation coefficient matrix, respectively. The m-index (with coefficient of 90 per cent) in Factor 1, a-index (with coefficient of 91 per cent) in Factor 2 and h-and h2-indexes (with coefficients of 93 per cent) in Factor 3 had the highest factor loadings. Correlation coefficients and related comparative diagrams showed that the mh-index is more accurate than the other nine variants in differentiating the scientific impact of researchers with the same h-index. Originality/value -As the studied variants could not satisfy all limits of the h-index, scientific society needs an index which accurately evaluates individual researcher's scientific output. As the mh-index has some advantages over the other studied variants, it can be an appropriate alternative for them.

show abstract

On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper

Cited by 32 publications

References 26 publications

Integrated impact indicators compared with impact factors: An alternative research design with policy implications

Integrated impact indicators compared with impact factors: An alternative research design with policy implications

A Framework for the Assessment of Research and Its Impacts

Multiple h-index: a new scientometric indicator

Contact Info

Product

Resources

About