How to improve the use of metrics

doi:10.1038/465870a

Cited by 29 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other side, there are also a lot of outputs that are crucial and this study does not take into account [ 20 ]. Moreover, it has been documented that publications, citations and patents are poor proxies of the outputs of S&T since to capture the essence of good science, evaluators of scientific activity should combine forces to create an open, sound and consistent system for measuring all the activities that make up academic productivity [ 13 , 20 – 25 ]. In this paper, we only consider publications and citations reported in ISI databases, and even though they are a valuable tool in policy studies addressing general issues regarding academic systems since they are objective measurements of the diffusion and impact of research, and allow us to determine the geographic origin of research and detect growth or erosion of countries´ scientific impact; they have important limitations, such as they do not take into account books, proceedings, local journals, etc., besides the count of citations does not consider misspellings [ 26 ].…”

Section: Discussionmentioning

confidence: 99%

The Scientific Impact of Developing Nations

et al. 2016

View full text Add to dashboard Cite

This paper analyzes science productivity for nine developing countries. Results show that these nations are reducing their science gap, with R&D investments and scientific impact growing at more than double the rate of the developed world. But this “catching up” hides a very uneven picture among these nations, especially on what they are able to generate in terms of impact and output relative to their levels of investment and available resources. Moreover, unlike what one might expect, it is clear that the size of the nations and the relative scale of their R&D investments are not the key drivers of efficiency.

show abstract

Section: Discussionmentioning

confidence: 99%

The Scientific Impact of Developing Nations

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Quantitative measures can and should still be used, but without understanding the context of individual researchers and their fields, no metric can solely provide evidence of impact, especially across disciplines and geographic scopes. Our findings suggest that university-wide assessments are overly burdensome, and experts agree that metrics should be used to complement or support qualitative assessment, with a greater concentration on professional judgment from other colleagues in the field or the department (Bergstrom, 2010;Muller, 2018).…”

Section: Discussionmentioning

confidence: 82%

“…Unfortunately, any metric that measures human performance is subject to manipulation and corruption (Muller, 2018). Opponents of metric culture typically do not condone metrics themselves or even the use of metrics, but rather the over-reliance on metrics; they argue that metric use in the absence of expert and qualitative assessment is irresponsible, inadequate, and even dangerous (Bergstrom, 2010;Edwards & Roy, 2016;Moustafa, 2016;Muller, 2018).…”

Section: Interpretations and Limitations Of Citation Impact Indicatorsmentioning

confidence: 99%

Faculty Perceptions of Research Assessment at Virginia Tech

Miles

Pannabecker

Kuypers

2020

joa

View full text Add to dashboard Cite

In the spring of 2019, survey research was conducted at Virginia Polytechnic Institute and State University (Virginia Tech), a large, public, Carnegie-classified R1 institution in southwest Virginia, to determine faculty perceptions of research assessment as well as how and why they use researcher profiles and research impact indicators. The Faculty Senate Research Assessment Committee (FSRAC) reported the quantitative and qualitative results to the Virginia Tech Board of Visitors to demonstrate the need for systemic, political, and cultural change regarding how faculty are evaluated and rewarded at the university for their research and creative projects. The survey research and subsequent report started a gradual process to move the university to a more responsible, holistic, and inclusive research assessment environment. Key results from the survey, completed by close to 500 faculty from across the university, include: a.) the most frequently used researcher profile systems and the primary ways they are used (e.g., profiles are used most frequently for showcasing work, with results indicating that faculty prefer to use a combination of systems for this purpose); b.) the primary reasons faculty use certain research impact indicators (e.g., number of publications is frequently used but much more likely to be used for institutional reasons than personal or professional reasons); c.) faculty feel that research assessment is most fair at the department level and least fair at the university level; and d.) faculty do not feel positively towards their research being assessed for the allocation of university funding.

show abstract

“…As Campbell (1979) warned, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (p. 85). Impact factor measurement not only restricts assessment of the quality of academic contributions to limited metrics, but it also “lures scientists to pursue high ranking first and good science second” (Bergstrom, 2010, p. 870). Also, a strong incentive is established for scientists, who under career-critical pressure to perform, to game the metrics that are meant to measure the quality of their work.…”

Section: Meta-scientific Issues In Judging Scholarly Workmentioning

confidence: 99%

Potential for Meta-Scientific Inquiry to Improve the Usefulness of HRD Research Outcomes for Practice

Passmore¹,

Chae

2019

Advances in Developing Human Resources

View full text Add to dashboard Cite

The Problem Meta-science, that is, science about science, is an application of scientific method to explain and control the methods and enterprise of science. The field of inquiry of meta-science emerged originally in reaction to problems with reproducibility and replicability of seemingly settled science. These same problems could affect research evidence conducted by human resource development (HRD) researchers and, as a result, could limit the applicability of HRD research outcomes to HRD practice. The Solution Discussed in this article are examples of meta-scientific issues and solutions that ultimately affect the transition of knowledge derived through research to practice in the field of HRD. Problems with the reproducibility and replication of research conducted in the field are detailed as are meta-scientific issues involved in judging the quality of scientific work, methodology-agnostic applications of meta-scientific inquiry, requirements for sharing data and code, and the need to let theory guide research. The Stakeholders Researchers would benefit from the considerations of meta-scientific concerns in the design, conduct, and reporting of research to improve the replicability and reproducibility of research outcomes. Knowledge of meta-scientific principles and cautions by adopters of research outcomes would inject healthy skepticism into decisions about whether research outcomes are worthy for application in practice or to guide future research.

show abstract

How to improve the use of metrics

Abstract: Since the invention of the science citation index in the 1960s, quantitative measuring of the performance of researchers has become ever more prevalent, controversial and influential. Six commentators tell Nature what changes might ensure that individuals are assessed more fairly.

Cited by 29 publications

References 8 publications

The Scientific Impact of Developing Nations

The Scientific Impact of Developing Nations

Faculty Perceptions of Research Assessment at Virginia Tech

Potential for Meta-Scientific Inquiry to Improve the Usefulness of HRD Research Outcomes for Practice

Contact Info

Product

Resources

About