Academic Evaluation: Universal Instrument? Tool for Development?

Bianco, Mariela; Gras, Natàlia; Sutz, Judith

doi:10.1007/s11024-016-9306-9

Cited by 57 publications

(49 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, some scholars have argued that non-mainstream journals offer a valuable communications channel for research that is neglected in mainstream journals (see debates between Kriege 1984 andMoravcsick 1987;Spinak 1996 andGarfield 1997;Beall 2015 andScielo 2015). The relevance of this debate to research policy is that it reveals a potential underestimation of the knowledge contained in non-mainstream journals by conventional research assessments and agendas (Bianco, Gras, & Sutz 2016). We address such a concern by examining the role of non-mainstream journals in scientific communication in the contributions and to give scientific recognition to the research community that uses it for publishing.…”

Section: Introductionmentioning

confidence: 99%

Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling

2017

View full text Add to dashboard Cite

Papers should be submitted to swps@sussex.ac.uk as a PDF or Word file. The first page should include: title, abstract, keywords, and authors' names and affiliations. The paper will be considered for publication by an Associate Editor, who may ask two referees to provide a light review. We aim to send referee reports within three weeks from submission. Authors may be requested to submit a revised version of the paper with a reply to the referees' comments to swps@sussex.ac.uk. The Editors make the final decision on the inclusion of the paper in the series. When submitting, the authors should indicate if the paper has already undergone peer-reviewing (in other series, journals, or books), in which case the Editors may decide to skip the review process. Once the paper is included in the SWPS, the authors maintain the copyright. AbstractIn many countries research evaluations confer high importance to mainstream journals, which are considered to publish excellent research. Accordingly, research evaluation policies discourage publications in other non-mainstream journals under the assumption that they publish low quality research. This approach has prompted a policy debate in low and middle-income countries, which face financial and linguistic barriers to access mainstream journals. A common criticism of the current evaluation practices is that they can hinder the development of certain topics that are not published in mainstream journals although some of them might be of high local relevance. In this article, we examine this issue by exploring the functions of non-mainstream journals in scientific communication. We interviewed researchers from agricultural sciences, business and management, and chemistry in Colombia on their reasons to publish in non-mainstream journals. We found that non-mainstream journals serve the following functions: 1) offer a space for initiation into publishing (training); 2) provide a link between articles in mainstream journals and articles read by communities with limited access to them (knowledge-bridging); 3) publish topics that are not well covered by mainstream journals (knowledge gap-filling). Therefore, publication in non-mainstream journals cannot be attributed only to 'low scientific quality' research. They also fulfil specific communication functions. These results suggest that research evaluation policy in low and middleincome countries should consider assigning greater value to non-mainstream journals given their role in bridging and disseminating potentially useful and novel knowledge.

show abstract

Section: Introductionmentioning

confidence: 99%

Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling

2017

View full text Add to dashboard Cite

show abstract

“…How research agendas are developed has always been a key question in the history, sociology, and economics of academic research (Bianco et al 2016). The forces that influence the development of research agendas are multifold and vary from popular research questions in a given socio-historical context to institutional dynamics relating to how the sciences formulate research puzzles (Merton 1942;Kuhn 1970).…”

Section: Peer Review and The Development Of Research Agendasmentioning

confidence: 99%

“…Indeed, private foundations, such as the Rockefeller Foundation, have always been significant players in the field of research funding (e.g., Smith 2009;Schneider 2015;Roelofs 2015). Bianco et al (2016) have argued that by means of research funding, promotion, and reputation, the peer review system and the development of research agendas are inevitably linked. We have a variety of funding sources-governments, universities, and foundations-that have their own goals as well as autonomy over how they organize evaluation and justify the distribution of resources (Bianco et al 2016).…”

Section: Peer Review and The Development Of Research Agendasmentioning

confidence: 99%

“…According to Bianco et al (2016), the academic evaluation system and the development of research agendas, conventional or unconventional, are inevitably linked by means of research funding. Whilst scholars have studied the evaluation of cross-disciplinary research proposals from the perspective of peer reviewers, they have paid comparatively little attention to how the performers of unconventional science experience evaluation practices (Langfeldt 2001;Langfeldt 2004;Lamont 2009;Mallard et al 2009;Lamont and Huutoniemi 2011;Huutoniemi 2012).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Power of Peer Review on Transdisciplinary Discovery

Mäkinen

2019

Science, Technology, & Human Values

View full text Add to dashboard Cite

This study examines the tension between the academic evaluation system and the development of unconventional research agendas. While scholars have studied the evaluation of research that crosses disciplinary boundaries from the perspective of peer reviewers, they have paid comparatively little attention to the experiences of the performers of unconventional science. This study asks how researchers develop unconventional research agendas to address a long-standing health problem and, in the process, make sense of the actions of a site visit committee organized to advise a foundation funding the project. This study develops a process narrative on the development of a specific transdisciplinary team proposing to study premature birth. The findings show that when the performers and the evaluators of unconventional science developed competing understandings of the research agenda, transdisciplinary discovery became limited as a particular research topic became taboo. Yet, the study also reveals how the performers of unconventional science challenged the power of the site visit committee by making it seem as though they followed the committee’s decisions. These findings raise questions about the role of private foundations as funders of academic research and the suitability of traditional evaluation procedures for assessing transdisciplinary discovery.

show abstract

“…In Latin America, perceptions of particularism in WoS have motivated the development of alternative journal databases such as Scielo and RedALyC(Chavarro, 2017, Chapter 3), which aim at giving visibility to research that is not usually covered by WoS. Similarly, many Latin American researchers have argued for more recognition of the knowledge produced in these journals(Packer & Meneghini 2007;Aguado-López et al 2014;Alperín 2014;Vessuri, Guédon, & Cetto 2014;Bianco, Gras, & Sutz 2016).…”

mentioning

confidence: 99%

To what extent is inclusion in the Web of Science an indicator of journal ‘quality’?

Chavarro

Ràfols

Tang

2018

Research Evaluation

View full text Add to dashboard Cite

The assessment of research based on the journal in which it is published is a widely adopted practice. Some research assessments use the Web of Science (WoS) to identify "high quality" journals, which are assumed to publish excellent research. The authority of WoS on journal quality stems from its selection of journals based on editorial standards and scientific impact criteria. These can be considered as universalistic criteria, meaning that they can be applied to any journal regardless of its place of publication, language, or discipline. In this article we examine the coverage by WoS of journals produced in Latin America, Spain, and Portugal. We use a logistic regression to examine the probability of a journal to be covered by WoS given universalistic criteria (editorial standards and scientific impact of the journal) and particularistic criteria (country, language, and discipline of the journal). We find that it is not possible to predict the inclusion of journals in WoS only through the universalistic criteria because particularistic variables such as country of the journal, its discipline and language are strongly related to inclusion in WoS. We conclude that using WoS as a universalistic tool for research assessment can disadvantage science published in journals with adequate editorial standards and scientific merit. We discuss the implications of these findings within the research evaluation literature, specifically for countries and disciplines not extensively covered by WoS. https://mc.manuscriptcentral.com/reseval Manuscripts submitted to Research Evaluation

show abstract

Academic Evaluation: Universal Instrument? Tool for Development?

Cited by 57 publications

References 28 publications

Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling

Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling

The Power of Peer Review on Transdisciplinary Discovery

To what extent is inclusion in the Web of Science an indicator of journal ‘quality’?

Contact Info

Product

Resources

About