Criteria for assessing research quality in the humanities: a Delphi study among scholars of English literature, German literature and art history

The future of research assessment in the humanities: bottom-up assessment procedures

“…All evaluative reports produced in 2010-2011 about the activity of all the research units in history and law from two French regions (Bretagne and Rhône-Alpes) have been scrutinized using methods and tools from corpus linguistics, in search of formulations allowing to understand how peer experts conceptualize and perceive quality in the activities and outputs they evaluate. While interviews conducted in parallel confirmed that experts from the two investigated fields diverge regarding their perceptions of quality-a finding which is in line with what other studies pointed out about the diversity of SSH disciplines when it comes to the conceptualization of research quality (see for example, Hug et al, 2013;Gogolin and Stumm, 2014;Lienhard et al, 2016)-it appears that reports do not echo these specificities adequately, since the main criteria they put forward are invariably the coherence of the research conducted in the evaluated unit and its productivity. It is not surprising, therefore, that the French SSH community found that the evaluation conducted by AERES was unsatisfactory on the whole and called for a radical modification of the exercise-a vow that was only very partially answered through the evolution of AERES towards HCERES 6 .…”

“…The literature so far describes the characteristics of SSH research in the following way: a) SSH research is interpretative, that is, humanities research is mainly text-and theory-driven and social sciences are more concept-driven, while the natural sciences set up their studies to answer specific questions and are progress-driven (MacDonald, 1994;Guetzkow et al, 2004;Lamont, 2009); it is reflective and introduces new perspectives in academia, by fostering discursive controversy and competing visions (Fisher et al, 2000;Hellqvist, 2010). With regard to the society, they bring a decisive contribution to the training of critical thinking as a prerequisite for democracy (Nussbaum, 2010) or to the critical examination of modern trends, such as technologisation (Luckmann, 2004); c) it is mainly individual (Finkenstaedt, 1990;Weingart et al, 1991), few publications are co-authored (Hemlin, 1996;Hellqvist, 2010) and research is often connected to the person conducting it (Hemlin and Gustafsson, 1996;Guetzkow et al, 2004); d) productivity is not that important for research performance in the SSH (Hemlin, 1993;Fisher et al, 2000;Hug et al, 2013); e) societal orientation is important, i.e. research is meant to influence society, direct interaction with society is part of SSH research (Weingart et al, 1991;Hellqvist, 2010;Hug et al, 2013); but f) the influence of society or other stakeholders outside of academia, such as external funding, on SSH research is evaluated negatively (Hemlin, 1993;Hug et al, 2013;Ochsner et al, 2013).…”

Humanities Scholars’ Conceptions of Research Quality

“…Because of the heavy workload required to respond to this questionnaire, it was administered to only a part of the sample (n = 180) scholars). The first round achieved a response rate of 28 % and resulted in a more refined catalogue of quality criteria, comprising 19 criteria specified by a total of 70 aspects (for a description of the method and the results, see Hug et al 2013). In the second Delphi round, which was administered to the whole sample N = 664), the scholars rated the aspects on a scale from 1 to 6 as to whether they agreed with a given statement.…”

Quality in the process of scientific production in accounting in Brazil

“…For example, surveys are used to judge quality in fields such as the humanities (Hug, Ochsner, & Daniel, 2013) and accounting (Brinn et al, 2001;Lowe & Locke, 2005;Lowensohn & Samelson, 2006;Northcott & Linacre, 2010). Another method is to judge the quality of evidence using surveys as the technique for the studies' data collection (Van der Stede, Young, & Chen, 2005).…”

“…Additionally, it is used to build evaluation frameworks for judging quality criteria (Elwyn et al, 2006). It is also used in other fields such as the humanities (Hug et al, 2013) (1) It is "subject to bias because the investigator limits the scope of the issue assessed by the panelists." Consequently, the issue in question is at least partially controlled by the researcher, and the consensus can thus be somewhat distorted;…”