A new approach to the analysis and evaluation of the research output of countries and institutions

Abstract

A plethora of bibliometric indicators is available nowadays to gauge research performance. The spectrum of bibliometric based measures is very broad, from purely size-dependent indicators (e.g. raw counts of scientific contributions and/or citations) up to size-independent measures (e.g. citations per paper, publications or citations per researcher), through a number of indicators that effectively combine quantitative and qualitative features (e.g. the h-index). In this paper we present a straightforward procedure to evaluate the scientific contribution of territories and institutions that combines size-dependent and scale-free measures. We have analysed in the paper the scientific production of 189 countries in the period 2006-2015. Our approach enables effective global and field-related comparative analyses of the scientific productions of countries and academic/research institutions. Furthermore, the procedure helps to identifying strengths and weaknesses of a given country or institution, by tracking variations of performance ratios across research fields. Moreover, by using a straightforward wealth-index, we show how research performance measures are highly associated with the wealth of countries and territories. Given the simplicity of the methods introduced in this paper and the fact that their results are easily understandable by non-specialists, we believe they could become a useful tool for the assessment of the research output of countries and institutions.

Factors Predicting the Scientific Wealth of Nations

“…Despite these problems, the mean citation rate combined with the percentage of papers that reach the top citation rate—the HQSI —still seems to be a sufficiently adequate measure of what can be called the scientific wealth of nations, better than the mean citation rate alone (cf. Docampo & Bessoule, 2019; Harzing & Giroud, 2014; Horta, 2018; King, 2004; May, 1997; Rousseau & Rousseau, 1998). Even though it has been argued that the traditional method of counting citations may not be the most accurate or optimal measure of scientific excellence (cf.…”

“…Just as gross domestic product (GDP) is a measure of economic wealth, the mean citation rate per scientific article could be used as a measure of the scientific wealth of nations (King, 2004;May, 1997). The link between economic and scientific wealth has been repeatedly demonstrated (Cimini et al, 2014;Cole & Phelan, 1999;Docampo & Bessoule, 2019;Gantman, 2012;Hatemi-J et al, 2016;King, 2004;Kumar et al, 2016;May, 1997;Mueller, 2016;Prathap, 2017;Rousseau & Rousseau, 1998;Vinkler, 2008Vinkler, , 2018, meaning that the richer the country, the more money is invested into research and development (R&D) activities, which, in return, is expected to increase the quality of scientific publications. Yet, the relationship between economic and scientific wealth is not as simple and straightforward as is often presented.…”

The link between countries’ economic and scientific wealth has a complex dependence on technological activity and research policy

“…Many studies have demonstrated that scientific wealth depends on economic wealth (Allik et al, 2020;Cimini et al, 2014;Cole & Phelan, 1999;de Moya-Anegón & Herrero-Solana, 1999;Docampo & Bessoule, 2019;Gantman, 2012;King, 2004;May, 1997;Mueller, 2016;Rahman & Fukui, 2003;Rousseau & Rousseau, 1998;Satish, 2021;Vinkler, 2018), and in recent times, e.g. last 50-60 years, all the scientific achievements that have been awarded with Nobel Prizes have come from rich and highly developed countries, with Federico Leloir from Argentina being perhaps the only exception to this general rule.…”

“…However, before drilling down into the details of what makes a research system more or less efficient, the economic factor must also be considered. Many studies have demonstrated that scientific wealth correlates with economic wealth (e.g., Allik et al, 2020;Cimini et al, 2014;Cole & Phelan, 1999;Docampo & Bessoule, 2019;Gantman, 2012;King, 2004;May, 1997;Mueller, 2016;Rahman & Fukui, 2003;Rousseau & Rousseau, 1998), which suggests that countries with medium or low GDP per capita might contribute very little to scientific progress. Such countries may have insufficient economic resources to fund and maintain a competitive research system or to develop the efficient functional structures required to perform research efficiently and compete successfully with richer countries in the advancement of science.…”

TeamTree analysis: A new approach to evaluate scientific production

“…Authorship on these articles credits contributions of individual team members with diverse expertise and skills [17][18][19][20][21], but choosing the best method to evaluate research, for example to identify potential experts, recruits and collaborators, remains a challenge [22]. Presently, the impact of individual contributors [23], journals [24], institutions and nations [25] is predominantly estimated based on citation counts of scientific articles (for reviews see [5,[26][27][28]). In a frequent scenario, a user interested in a specific topic queries a bibliographic database, scrutinizes the resulting list of relevant publications and learns readily about scientific advances.…”