Communicating and compromising on disciplinary expertise in the peer review of research proposals

Abstract

This paper analyses peer review deliberations in four evaluation panels that differ in terms of scope and disciplinary heterogeneity. Based on evaluation reports and discussions with panel members, it illustrates a variety of ways in which reviewers bridge their epistemological differences and achieve consensus on the quality of research proposals. The analysis demonstrates that peer review panels are forums in which communication across disciplinary boundaries occurs and interdisciplinary judgments arise. At the same time, disciplinary gate-keeping and incommensurabilities may set limits on such communication. The comparison of deliberative processes sheds light on how collective judgments are shaped and constrained by the disciplinary set-up of the panels in which the reviewers operate and in which the intersubjective dynamics of the deliberations unfold. Based on these findings, the paper considers conditions that may enhance disciplinary interaction and complementary judgments in the peer review of proposals, and thereby expands the prospects for interdisciplinary research.This study is concerned with the internal functioning of peer review, the practice through which scholarly work is evaluated by those with demonstrated competence. It analyses the ways in which peer review panels produce consensual judgments on the quality of research proposals and how reviewers are able to bridge their epistemological differences in this process. The topic is of interest, especially for those of us who are concerned with the status and fate of less established forms of inquiry -most typically,

Legitimate judgment in art, the scientific world reversed? Maintaining critical distance in evaluation

“…Just as definitions of value vary across contexts, so do the norms for what constitutes legitimate evaluation (Huutoniemi, 2012). Knorr-Cetina (1999) emphasizes the importance of context in her concept of epistemic cultures: the practical, symbolic, and material settings that structure what types of knowledge are produced and valued.…”

Interdisciplinarity as Academic Accountability: Prospects for Quality Control Across Disciplinary Boundaries

“…as it unfolds in the deliberation among reviewers. A comparative case study was conducted on four review panels in environmental and social sciences 2 that differed in terms of scope and disciplinary heterogeneity (Huutoniemi 2012b). The latter data consist of a diverse set of research proposals (n = 109), evaluation reports (n = 329), and interviews with panel members (n = 18) and funding officers.…”

“…What is relevant here is the finding that the composition of a panel creates its own particular sphere of academic control and reciprocal accountability, which influences the panellists' tendency to build on and scrutinize each others' judgements (Huutoniemi 2012b). While peer reviewers who conduct evaluations as individuals are only loosely accountable for their judgements, a new sphere of accountability arises between peer reviewers in the panel setting: experts judge each other's standards and behaviour as much as they judge the proposals (see Hirschauer 2010;Lamont 2009).…”

“…In interdisciplinary panels, this often covers epistemic positions behind judgements in addition to the technical validity of judgements as such. I have previously demonstrated this by analyzing and comparing the discursive practices through which the members of selected panel deliberations were able to reach consensus (Huutoniemi 2012b).…”

“…This variation points to two important factors for interdisciplinary accountability to arise (see Huutoniemi 2012b). One is the mix of experts in a panel, in terms of the overlap in the panellists' competencies.…”