Fairness as Appropriateness

Abstract

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, consensus, and the norm of universalism, but neglected the content of evaluation and how epistemological differences affect perception of fairness in decision-making. The normative literature suggests that evaluators should overcome their epistemological differences by "translating" their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates in panels, and more specifically about how agreements are reached in the face of epistemological diversity. Drawing on 81 interviews with panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that: 1) Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles. 2) Although the comprehensive style is favored, there is considerable diversity in the epistemological styles used in the panels we studied; 3) Peer reviewers define a fair-decision making process as one in which panelists engage in "cognitive contextualization," that is, use epistemological styles most appropriate to the field or discipline of the proposal under review.These findings challenge the normative literature that associates procedural fairness with the use of generalizable criteria of evaluation. 3 Fairness as Appropriateness: Negotiating Epistemological Differences in Peer ReviewTheoretical and methodological approaches are objects of continuous and frequently divisive debates in the social sciences and the humanities (Bourdieu, Chamboredon, and Passeron 1968;Merton 1972;Somers 1996;Steinmetz 2005). When textbooks and specialized publications refer to differences in theoretical and methodological approaches, they often frame them as irreconcilable epistemological styles, stressing incompatible elements. By "epistemological styles" we refer to scholars' preferences for particular theoretical styles (ways of understanding how to build theories and how to accumulate knowledge) and methodological styles (methods of proving, and belief in the very possibility of proving, theories (Knorr-Cetina 1999)). The diversity of theoretical styles ranges from the view that authors should acknowledge how the formulation of their theoretical orientation is shaped by their own social location, identity, and political orientation (DeVault 1999;Smith 1990), to the view that theories emerge from the observation of new evidence in light of existing explanations, without being affected by who the researcher is or how she apprehends her object (Nagel 1961). Regarding methodological styles, authors such as Nagel (1961), Ragin (1987), Singleton and Straits (1999), Stinchcombe (2005) and Tilly (1994) hav...

The Power of Peer Review on Transdisciplinary Discovery

“…The tension between the academic evaluation system and the development of unconventional research agendas is irrefutable. Prior work has shed light on how peer reviewers make decisions about cross-disciplinary research proposals in the presence of epistemological differences (Lamont 2009;Lamont and Huutoniemi 2011;Mallard et al 2009;Huutoniemi 2012). When researchers take on the role of an evaluator, even if the object of evaluation is in some way foreign to them, quality is something they claim to "know" through their prior experience (Lamont 2009;Langfeldt 2004).…”

“…According to Bianco et al (2016), the academic evaluation system and the development of research agendas, conventional or unconventional, are inevitably linked by means of research funding. Whilst scholars have studied the evaluation of cross-disciplinary research proposals from the perspective of peer reviewers, they have paid comparatively little attention to how the performers of unconventional science experience evaluation practices (Langfeldt 2001;Langfeldt 2004;Lamont 2009;Mallard et al 2009;Lamont and Huutoniemi 2011;Huutoniemi 2012). To advance knowledge on the topic, this study asks how researchers develop unconventional research agendas to address a long-standing health problem and, in the process, make sense of the actions of a site visit committee organized to assess progress and advise a research funder.…”

How Quality Is Recognized by Peer Review Panels: The Case of the Humanities

“…The evaluative technique imposed by the funding agency thus influences the behaviour of panelists. Moreover, the customary rules of methodological pluralism and cognitive contextualism (Mallard et al 2009) are more salient in the humanities and social science panels than they are in the pure and applied science panels, where disciplinary identities may be unified around the notion of scientific consensus, including the definition of shared indicators of quality. Finally, a concern for the use of consistent criteria and the bracketing of idiosyncratic taste is more salient in the sciences than in the social sciences and humanities, due in part to the fact that in the latter disciplines evaluators may be more aware of the role played by (inter)subjectivity in the evaluation process.…”

The ethics of scholarly peer review: a review of the literature

“…79 When the peer-review process becomes interdisciplinary, some scholars have discovered epistemological conflicts of interest. Mallard et al interviewed 81 panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities 80 and identified four epistemological styles in their arguments for or against funding a proposal. Conflicts of interest arise when a reviewer evaluates a proposal from an epistemological point of view that is inconsistent with the author's.…”