A critical evaluation of expert survey‐based journal rankings: The role of personal research interests

Abstract

By using the data from two recent survey‐based rankings of knowledge management / intellectual capital and eHealth journals, this study tests the impact of personal research interests of journal raters on their ranking scores. The rationale is that raters assign higher scores to journals that cater to their area of expertise because they are more familiar with them. The results indicate the existence of raters’ bias toward the journals focusing on their preferred areas of interest, but this bias does not uniformly apply across all research topics. In some subdomains, such as intellectual capital, this bias may be very strong, whereas in others, such as soft‐side knowledge management research, it may be nonexistent. Although management eHealth researchers rate management‐focused journals higher than their clinical‐centered counterparts, this bias does not exist among scholars favoring clinical topics. While this limitation is not fatal, the results from expert‐survey journal ranking studies should be interpreted with caution.

Where Do Nonprofit and Civil Society Researchers Publish? Perceptions of Nonprofit Journal Quality

“…Second, solely relying on the impact factor to rank journals or, by extension, the quality of an author runs the risk of assuming that only journals with high impact factors are publishing high-quality research and researchers (Adler & Harzing, 2009;Aguinis et al, 2019;Singh, Haddad, & Chow, 2007). Third, as Serenko and Bontis (2018) argued, "It is extremely difficult to deliberately influence the journal quality perceptions of a large group of independent scholars, whereas citations may be dramatically boosted in the short term by means of questionable practices such as forced citations or excessive self-citations" (p. 749). As such, the stated preference approach is considered more suitable for the purposes of our study, since developing a revealed preference journal ranking may not only be difficult to achieve (given the interdisciplinary nature of nonprofit and civil society studies), but it is also not desirable given the well-known limitations of these rankings.…”

“…By inviting nonprofit and civil society scholars to free write journal names instead of presenting them with a predetermined list, we are able to more accurately account for the journal outlets that exist within the field. Indeed, using this approach we are able to capture journals that may not (yet) have an impact factor as well as those that do not aim to achieve one (Serenko & Bontis, 2018). In general, the journals identified in this study are, on average, newer and more specialized in substantive content area than are journals with an impact factor (Adler & Harzig, 2009).…”

“…Many studies exploring perceptions of journal quality have used predetermined lists that respondents are asked to rank (Catling, Mason, & Upton, 2009;Meggs et al, 2017;Serenko & Bontis, 2018). In order to be able to rank journals from a list, however, it is important that respondents have familiarity with the journals that are listed (Walters, 2017).…”

“…Moreover, the stated preference approach of capturing perceived journal quality used in this study can be prone to bias (Tahai & Meyer, 1999). For instance, scholars may give preference to journals they are familiar with (Walters, 2017) as well as journals that influence their personal research interests (Serenko & Bontis, 2018). Future research, therefore, should include measures of familiarity to disentangle and control for the influence of personal research interests.…”

Algorithm for Producing Rankings Based on Expert Surveys

“…It is difficult to obtain objective data on corruption, so instead, CPI is created on the basis of expert opinion [3,4]. Although the views of the experts whose input is used for the CPI are based on imperfect information and carry a risk of bias [5][6][7], the resulting index is nonetheless a useful tool to inform the political choices of citizens, guide companies in their investment decisions, and help governments understand how their level of corruption is seen [2,8].…”