Notions of research quality are contextual in many respects: they vary between fields of research, between review contexts and between policy contexts. Yet, the role of these co-existing notions in research, and in research policy, is poorly understood. In this paper we offer a novel framework to study and understand research quality across three key dimensions. First, we distinguish between quality notions that originate in research fields (Field-type) and in research policy spaces (Space-type). Second, drawing on existing studies, we identify three attributes (often) considered important for 'good research': its originality/novelty, plausibility/reliability, and value or usefulness. Third, we identify five different sites where notions of research quality emerge, are contested and institutionalised: researchers themselves, knowledge communities, research organisations, funding agencies and national policy arenas. We argue that the framework helps us understand processes and mechanisms through which 'good research' is recognised as well as tensions arising from the co-existence of (potentially) conflicting quality notions. 3Co-existing Notions of Research Quality dimensions of our framework, encompassing: (i) types of quality notions; (ii) attributes of quality; and (iii) sites where notions are established and institutionalised. Third, we combine these three dimensions into an overall framework to study and understand research quality notions. We then discuss ways in which this proposed approach stands to change the research agenda.

Research funding is an important factor for public science. Funding may affect which research topics get addressed, and what research outputs are produced. However, funding has often been studied simplistically, using top-down or system-led perspectives. Such approaches often restrict analysis to confined national funding landscapes or single funding organizations and instruments in isolation. This overlooks interlinkages, broader funding researchers might access, and trends of growing funding complexity. This paper instead frames a ‘bottom-up’ approach that analytically distinguishes between increasing levels of aggregation of funding instrument co-use. Funding of research outputs is selected as one way to test this approach, with levels traced via funding acknowledgements (FAs) in papers published 2009–18 by researchers affiliated to Denmark, the Netherlands or Norway, in two test research fields (Food Science, Renewable Energy Research). Three funding aggregation levels are delineated: at the bottom, ‘funding configurations’ of funding instruments co-used by individual researchers (from single-authored papers with two or more FAs); a middle, ‘funding amalgamations’ level, of instruments co-used by collaborating researchers (from multi-authored papers with two or more FAs); and a ‘co-funding network’ of instruments co-used across all researchers active in a research field (all papers with two or more FAs). All three levels are found to include heterogenous funding co-use from inside and outside the test countries. There is also co-funding variety in terms of instrument ‘type’ (public, private, university or non-profit) and ‘origin’ (domestic, foreign or supranational). Limitations of the approach are noted, as well as its applicability for future analyses not using paper FAs to address finer details of research funding dynamics.

No abstract

We describe the development and testing of a framework to characterize researchers individually (a profile) and in aggregate (as types) at the first stage, baseline step of a controlled, two-stage study of public research funding agency impacts. Our framework characterizes researcher attitudes and attributes, and conditions and opportunities experienced, addressing: 'demographic' factors; researcher 'approach'; and 'standing' (organizational career, knowledge community career, and local and national research environment aspects). This integrated demographic-approach-standing (DAS) framework is tested using a survey of 184 applicants to the inaugural, 2007 call for the European Research Council (ERC) Starting Grants (StG). Successful applicants are characterized to match-pair with a control group of quality-screened unsuccessful applicants. Given the inherent difficulty to identify in advance the 'frontier' researchers the ERC aims to fund with its StG, we characterize 'frontier-potential' factors that might lead researchers later to become regarded as frontier. We develop researcher types using several of our framework elements: researcher intellectual field mobility; novelty and risk-taking; independence; output productivity; and local research workplace standing. We find a variety of grantee types, but primarily not yet independent researchers whose 'standing' could be impacted upon by the early-career StG scheme. Lastly, we suggest impact pathways between types we could capture following a second stage survey, and discuss limitations to our framework including revising it to characterize better local research environment aspects.

The current range and volume of research evaluation-related literature is extensive and incorporates scholarly and policy/practice-related perspectives. This reflects academic and practical interest over many decades and trails the changing funding and reputational modalities for universities, namely increased selectivity applied to institutional research funding streams and the perceived importance of university rankings and other reputational devices. To make sense of this highly diverse body of literature, we undertake a critical review of over 350 works constituting, in our view, the ‘state-of-the-art’ on institutional performance-based research evaluation arrangements (PREAs). We focus on PREAs because they are becoming the predominant means world-wide to allocate research funds and accrue reputation for universities. We highlight the themes addressed in the literature and offer critical commentary on the balance of scholarly and policy/practice-related orientations. We then reflect on five limitations to the state-of-the-art and propose a new agenda, and a change of perspective, to progress this area of research in future studies.