ISRIA statement: ten-point guidelines for an effective process of research impact assessment

Abstract

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school’s learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders’ needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school’s faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

Reporting health research translation and impact in the curriculum vitae: a survey

“…We defined research translation activities as the diffusion, dissemination, and application of knowledge that researchers undertake once the findings from a study are available [13]. Impact is the demonstrable contribution that research makes to the economy and the society, beyond academic contributions (e.g., publications and presentations) [22]. Indicators of impact can be instrumental (e.g., development or changes in policy, practice, services, legislation, behavior), conceptual (e.g., contribute to understanding of policy issues, reframing debates), and capacity building (e.g., technical and personal skill development).…”

Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada

“…The importance of identifying and being explicit about the objective or purpose of the impact assessment is emphasized by the International School of Research Impact Assessment (ISRIA), of which Alberta Innovates was a co-founder. ISRIA builds on the work of RAND (Guthrie et al, 2013) that is reiterated in the 10-point guideline for an effective process of research impact assessment published by Adam et al (2018). In practice, these purposes are commonly referred to as the "4As" of impact assessment and are used by Alberta Innovates as part of its six-block protocol for implementing fit-for-purpose frameworks:…”

“…Best practice guidance on communicating impact to diverse stakeholders requires integrating quantitative impact information with narratives (Adam et al, 2018). The narratives provide additional qualitative information that help contextualize the impact achieved by researchers.…”

Extending the concept of research impact literacy: levels of literacy, institutional role and ethical considerations

“…In the last cycle (REF, 2014), impact accounted for 20% of the allocation, which has been increased to 25% for the current (2021) round. Impact assessment is an increasingly mechanised means to support goals that include accountability, allocation, advocacy and analysis (Adam et al, 2018), and so it is imperative that those within the research sector critically understand how research can lead to change. Conversely in countries where government assessment is not a dominant feature, impact is instead driven by institutional, researcher or funder goals ('mission driven'), with Research Impact Canada and the US based National Alliance for Broader Impacts exemplars of networks working within this mission brief.…”