Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories

Aksnes, Dag W.; Langfeldt, Liv; Wouters, Paul

doi:10.1177/2158244019829575

Cited by 721 publications

(543 citation statements)

References 117 publications

(125 reference statements)

Supporting

Mentioning

421

Contrasting

Unclassified

Order By: Relevance

“…In this last section, we compare our results to criteria found in studies on research quality and guidelines of funding agencies, discuss our results in relation to two normative positions, the fairness doctrine and the ideal of impartiality, and present limitations and avenues for future research. Aksnes et al (2019) argue from a context-independent perspective that originality, scientific value, societal value, and plausibility/soundness are the key dimensions of research quality and that each of these four dimensions includes a variety of aspects, which may be context-dependent. According to our analysis, these four dimensions are clearly present in grant peer review.…”

Section: Resultsmentioning

confidence: 99%

“…While originality, academic relevance, and extra-academic relevance were mostly used to assess the aims and the expected results of the proposed project, appropriateness, rigor, coherence/justification, and quality were mostly used to assess entities pertaining to the research process (e.g., data, theory, method, analysis). In addition to the four dimensions of Aksnes et al (2019), our synthesis has shown that three other dimensions are important in assessing the merit of grant proposals. The first dimension, quality of description, assesses how the proposed project and information about the applicant are reported and presented in terms of the criteria clarity and completeness.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a 'psychological' (motivation, traits) and a 'sociological' (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

show abstract

“…In this last section, we compare our results to criteria found in studies on research quality and guidelines of funding agencies and then discuss our results in relation to two normative positions, the fairness doctrine and the ideal of impartiality. Aksnes, Langfeldt, & Wouters (2019) argue from a context-independent perspective that originality, scientific value, societal value, and plausibility/soundness are the key dimensions of research quality and that each of these four dimensions includes a variety of aspects, which may be context-dependent. According to our analysis, these four dimensions are clearly present in grant peer review.…”

Section: Resultsmentioning

confidence: 99%

“…data, theory, method, analysis). In addition to the four dimensions of Aksnes et al (2019), our synthesis has shown that three other dimensions are important in assessing the merit of grant proposals. The first dimension, quality of description, assesses how the proposed project and information about the applicant are reported and presented in terms of the criteria clarity and completeness.…”

Section: Resultsmentioning

confidence: 99%

Criteria for assessing grant applications: A systematic review

Hug¹,

Aeschbach²

2019

Preprint

View full text Add to dashboard Cite

Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use in assessing grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce the Scriven Model, which separates each criterion into an evaluated entity (i.e. the object of the evaluation) and an evaluation criterion (i.e. the dimension along which an entity is evaluated). In total, this synthesis includes 12 studies. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities as well as the relations between them. Based on a network analysis, we propose a conceptualization that groups the identified evaluation criteria and evaluated entities into aims, means, and outcomes. We compare our results to criteria found in studies on research quality and guidelines of funding agencies. Since peer review is often approached from a normative perspective, we discuss our findings in relation to two normative positions, the fairness doctrine and the ideal of impartiality. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.

show abstract

“…Reasons for self-citation include: "the cumulative nature of individual research, the need for personal gratification, or the value of self-citation as a rhetorical and tactical tool in the struggle for visibility and scientific authority" (Fowler and Aksnes 2007). Such reasons imply behaviors that conform both to Mertonian norms and also to the social constructivist interpretation of citation theory (Merton 1942;Kaplan 1965;Moravcsik and Murugesan 1975;Gilbert 1977;Cozzens 1989;Bonzi and Snyder 1991;White 2001;Bornmann and Daniel 2008;Davis 2009;Erikson and Erlandson 2014;Bornmann 2018, 2019;Aksnes et al 2019). Small (2004) offered a synthesis of motives for citation, in which citations serve a dual function as "vehicles of peer recognition and constructed symbols for specific original achievements."…”

Section: Studies Of Self-citationmentioning

confidence: 99%

How much is too much? The difference between research influence and self-citation excess

Szomszor¹,

Pendlebury²,

Adams

2020

Scientometrics

View full text Add to dashboard Cite

Citations can be an indicator of publication significance, utility, attention, visibility or short-term impact but analysts need to confirm whether a high citation count for an individual is a genuine reflection of influence or a consequence of extraordinary, even excessive, self-citation. It has recently been suggested there may be increasing misrepresentation of research performance by individuals who self-cite inordinately to achieve scores and win rewards. In this paper we consider self-referencing and self-citing, describe the typical shape of self-citation patterns for carefully curated publication sets authored by 3517 Highly Cited Researchers and quantify the variance in the distribution of self-citation rates within and between all 21 Essential Science Indicators' fields. We describe both a generic level of median self-referencing rates, common to most fields, and a graphical, distribution-driven assessment of excessive self-citation that demarcates a threshold not dependent on statistical tests or percentiles (since for some fields all values are within a central 'normal' range). We describe this graphical procedure for identifying exceptional self-citation rates but emphasize the necessity for expert interpretation of the citation profiles of specific individuals, particularly in fields with atypical self-citation patterns. , for their advice and suggestions during the development of this work. We also thank an anonymous referee for their comments regarding the 'typicality' of highly-cited researchers.

show abstract

Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories

Cited by 721 publications

References 117 publications

Criteria for assessing grant applications: a systematic review

Criteria for assessing grant applications: a systematic review

Criteria for assessing grant applications: A systematic review

How much is too much? The difference between research influence and self-citation excess

Contact Info

Product

Resources

About