‘Environment’ Submissions in the UK's Research Excellence Framework 2014

Abstract

There has been much debate about university research assessment exercises. In the UK, a major element of the 2014 Research Excellence Framework (REF2014) has been the research environment. Here we analyse 98 REF2014 ‘environment’ submissions in Business and Management Studies. We explore whether there are distinctive language‐related differences between submissions of high and low ranked universities and conclude that submission writers have a strong incentive to exaggerate strengths and conceal problems. In addition, innate biases such as the ‘halo’ and ‘velcro’ effects may distract the attention of assessors from a submission's strengths and weaknesses, since they are likely to influence their pre‐existing impressions. We propose several changes to improve how environment is evaluated. We also argue that the research environment would be more likely to be enhanced if the number of outputs submitted in future was an average of two and a maximum of four per academic, rather than the maximum of five currently being considered.

History, Evolution and Future of Big Data and Analytics: A Bibliometric Analysis of Its Relationship to Performance in Organizations

“…The style of writing the papers may differ from function to function, which can, for example, suggest that certain writing styles are more frequent in one function over the other (e.g. Thorpe et al ., ) and hinder the dissemination of findings (e.g. methodological advancements clusters vs. mainstream management clusters).…”

Semantic tone of research ‘environment’ submissions in the UK’s Research Evaluation Framework 2014

“…In the example just mentioned, the GPA is 3.25 ([4 x 0.50] + [3 x 0.375] + [1 x 0.125]). As with Thorpe et al (2017), we ranked the submissions according to their GPA score for 'Environment' (from best to worst) and then allocated them to quartile groups, designated as Q1, Q2, Q3 and Q4 (see Table 1). To resolve deadlocks arising from an equal GPA, the higher(est) rank was assigned to institutions submitting the larger(est) number of FTE staff.…”

“…Our study extends the work of Thorpe et al (2017) by using computer assisted text analysis (CATA) techniques to explicitly explore the semantic tone of research 'Environment' submissions made to REF2014. CATA techniques have many potential benefits.…”

“…His analysis on how the writing of the submissions was shaped, and the effect of particular words, was based principally on a close reading of the proposals and related documents, and interviews with proposal writers. Thorpe et al (2017) adopted a similar approach in applying an 'impression management lens' and close reading analysis to scrutinise how university submissions to the UK Research Excellence Framework 2014 (REF2014) were crafted, and whether stylistic differences in submissions could be linked to published outcomes.…”

“…4 Our use of DICTION software to analyse the semantic tone of 'Environment' submissions contributes to development of a deeper understanding of research evaluation methods and outcomes, including the possibility of better appreciating the incidence and features of 'gameplaying by wording.' There has been a relatively low level of scholarly scrutiny of the 'Environment' component of REF2014 (exceptions are Pidd and Broadbent [2015]; Mellors-Bourne, Metcalfe and Gill [2017]; and Thorpe et al [2017]). This is surprising for two reasons.…”