How Much Does Journal Reputation Tell Us About the Academic Interest and Relevance of Economic Research? Empirical Analysis and Implications for Environmental Economic Research

Abstract

Unlike in other disciplines, research output in economics is commonly measured based on the disciplinary reputation of the journals in which an author has published. Here, I examine how much output measures based on journal reputation tell us about the academic interest and relevance of economic papers as measured by frequency of citation. Using data from the 2008 Handelsblatt ranking of economists in German speaking countries and interdisciplinary citation data from the Web of Science, I find that researcher scores based on journal reputation explain only about 30 percent of the variation (variance) in article citations. When the top 10 (20) percent of the researchers according to journal reputation scores are excluded, the percentage of explained variation in citation frequency drops to 8 (3) percent. Furthermore, using environmental economics journals as an example, I show that the traditional output measures strongly discourage applied and interdisciplinary economic research. The findings confirm that the traditional output measures provide incentives for narrow economic work even if that work is of interest to only few other researchers. Responsible hiring committees and funding institutions should take these problems seriously and re-consider existing standards in the evaluation of economic research.

Messung der akademischen Forschungsleistung in den Wirtschaftswissenschaften: Reputation vs. Zitierhäufigkeiten

“…Bei jenem Ranking betrug die erklärte Varianz in analogen Analysen (analog zum 1., 3. und 4. Modell in Tabelle 1) 31, 8 und 3 Prozent (Schläpfer 2010). Die Punktwerte und Zitationshäufigkeiten der ersten 50 Forschenden im Handelsblatt ‐Ranking 2008 sind im aufgeführt 9…”

Physical Education Pedagogy Faculty Perceptions of Journal Quality

“…It has been suggested that peer assessment among scholars in a field provides an evaluation of perceptions of journal quality, journal use, and is an indicator of general value within a field. This technique of surveying scholars has been used in a number of other academic fields, including medicine (Yue, Wilson, & Boller, 2007), economics (Schläpfer, 2010), psychology (Catling, Mason, & Upton, 2009), sport management (Shilbury & Rentschler, 2007), and other areas (e.g., Extejt & Smith, 1990;Nederhof & Zwaan, 1991;Rousseau, 2008). Peer assessment provides another dimension of journal quality and can be performed within an academic specialty.…”

Measurements and metrics in small technology and knowledge entrepreneurship

“…This change is creating exceptional opportunity [60,64]. This reality is not addressed by traditional roadmapping techniques.…”

“…The new technologies used in pharmaceutical innovation development are disruptive in nature [64,65]. This has created great interest in the sector by firms and economic development professionals who are not currently involved in the pharmaceutical sector since they believe they can leapfrog existing and formerly essential infrastructure.…”

“…Attempts to address concerns regarding issues of quality and quantity have resulted in a number of alternatives to evaluation, including citation counts, 4,[57][58][59] journal impact scores, 52, [60][61][62][63] Hirsch Indices, 52,64,65 journal reputation studies, 48,63,[66][67][68][69] and Internet citation rates. 70,71 The result has been a proliferation of measures to assess research, researchers, research outlets, and the locations in which the research occurs.…”