Cognitive Distances between Evaluators and Evaluees in Research Evaluation: A Comparison between Three Informetric Methods at the Journal and Subject Category Aggregation Level

Abstract

This article compares six informetric approaches to determine cognitive distances between the publications of panel members (PMs) and those of research groups in discipline-specific research evaluation. We used data collected in the framework of six completed research evaluations from the period 2009-2014 at the University of Antwerp as a test case. We distinguish between two levels of aggregation-Web of Science Subject Categories and journals-and three methods: while the barycenter method (2-dimensional) is based on global maps of science, the similarity-adapted publication vector (SAPV) method and weighted cosine similarity (WCS) method (both in higher dimensions) use a full similarity matrix. In total, this leads to six different approaches, all of which are based on the publication profile of research groups and PMs. We use Euclidean distances between barycenters and SAPVs, as well as values of WCS between PMs and research groups as indicators of cognitive distance. We systematically compare how these six approaches are related. The results show that the level of aggregation has minor influence on determining cognitive distances, but dimensionality (two versus a high number of dimensions) has a greater influence. The SAPV and WCS methods agree in most cases at both levels of aggregation on which PM has the closest cognitive distance to the group to be evaluated, whereas the barycenter approaches often differ. Comparing the results of the methods to the main assessor that was assigned to each research group, we find that the barycenter method usually scores better. However, the barycenter method is less discriminatory and suggests more potential evaluators, whereas SAPV and WCS are more precise.

Bibliometric and Visualized Analysis of China's Smart Grid Research 2008–2018

“…The WOS records were chosen as they contain the most comprehensive and influential journals, which have scientific robustness. Therefore, they have become the choice of many scholars for literature measurement (Rahman et al, 2017 ; Olawumi and Chan, 2018 ; Yu and He, 2020 ).…”

Excellence or Misconduct: How the Visibility of Team Leaders Impacts the Research Project Competition in the Republic of Moldova?

“…All these examples confirm not just the shortcomings of the project evaluation methodology, but also the fact that experts were appointed by people with inadequate knowledge of the call priorities, while appointed experts didn't possess skills suitable for the evaluation of all proposals. The content of the evaluation forms lead us to believe that many evaluators are not experts in their fields and are poorly qualified, resulting in corresponding consequences, as in similar situations referenced in the literature [9,23,24,35]. These results caused distrust in the objectivity of the evaluation process, suspicions of premeditated score reduction for some projects and favoring of others, including the "old boys network".…”

“…When discussing individual researchers, grants and project proposals, it seems that the correlation depends on some factors, including the type and purpose of the peer review. A number of studies have demonstrated a strong correlation between bibliometric indicators and several entities, such as 1) the results of fellowship applications [10], 2) peer review of research programs in the Netherlands, both in physics [23] and chemistry [16], 3) the peer review results of the candidates for the postdoctoral programs in Switzerland [24], 4) selection process for two European molecular biology programs -Long-term Fellowship and the Young Investigator programs [10], 5) peer review of library and communication science researchers [12], 6) evaluation of individual researchers [25], 7) reviewers' ratings for Spanish researchers [1], 8) selection of junior and senior researchers [26], 9) the grant allocation of the Netherlands Research Council [2].…”

Távolság a tudományos együttműködési hálózatokban

“…A citációs adatbázisokban az adatrekordok is az egyes publikációkra vonatkoznak. A különböző vizsgálatok azonban különböző aggregációs szinteken történhetnek, mint például a szerzők (Melin és Persson, 1996;Zhao és Strotmann, 2008a, 2008b, a cikkek (Luukkonen et al, 1993;Ahlgren és Jarneving, 2008;Hoekman et al, 2010) vagy a folyóiratok (Vida, 2016;Rahman et al, 2017) szintje. E szintek közül tehát alapszintnek a cikkek szintje tekinthető, a legtöbb elemzés is erre vonatkozik.…”