Evaluating impact from research: A methodological framework

Reed, Mark; Ferré, Marie; Martín-Ortega, Julia; Blanche, Rachel; Lawford-Rolfe, R.; Dallimer, Martin; Holden, Joseph

doi:10.1016/j.respol.2020.104147

Cited by 137 publications

(97 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, these conflicting impacts may apply to the same individual or across different stakeholders. Different aspects of research impact may be more or less important in different contexts and the relative value of these different impacts will need to be considered to enable meaningful evaluation [42][43][44].…”

Section: Discussionmentioning

confidence: 99%

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

Newington

Wells

Adonis

et al. 2021

BMC Health Serv Res

View full text Add to dashboard Cite

Background There are increasing opportunities for healthcare professionals outside medicine to be involved in and lead clinical research. However, there are few roles within these professions that include time for research. In order to develop such roles, and evaluate effective use of this time, the range of impacts of this clinical academic activity need to be valued and understood by healthcare leaders and managers. To date, these impacts have not been comprehensively explored, but are suggested to extend beyond traditional quantitative impact metrics, such as publications, citations and funding awards. Methods Ten databases, four grey literature repositories and a naïve web search engine were systematically searched for articles reporting impacts of clinical academic activity by healthcare professionals outside medicine. Specifically, this did not include the direct impacts of the research findings, rather the impacts of the research activity. All stages of the review were performed by a minimum of two reviewers and reported impacts were categorised qualitatively according to a modified VICTOR (making Visible the ImpaCT Of Research) framework. Results Of the initial 2704 identified articles, 20 were eligible for inclusion. Identified impacts were mapped to seven themes: impacts for patients; impacts for the service provision and workforce; impacts to research profile, culture and capacity; economic impacts; impacts on staff recruitment and retention; impacts to knowledge exchange; and impacts to the clinical academic. Conclusions Several overlapping sub-themes were identified across the main themes. These included the challenges and benefits of balancing clinical and academic roles, the creation and implementation of new evidence, and the development of collaborations and networks. These may be key areas for organisations to explore when looking to support and increase academic activity among healthcare professionals outside medicine. The modified VICTOR tool is a useful starting point for individuals and organisations to record the impact of their research activity. Further work is needed to explore standardised methods of capturing research impact that address the full range of impacts identified in this systematic review and are specific to the context of clinical academics outside medicine.

show abstract

Section: Discussionmentioning

confidence: 99%

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

Newington

Wells

Adonis

et al. 2021

BMC Health Serv Res

View full text Add to dashboard Cite

show abstract

“…As crucial as it is to meet the knowledge quality criteria in order to bridge the gap between science and practice, other process-oriented values, such as "mutual interest in longer-term collaboration" (Wall et al, 2017), can contribute to that objective in the longer term, and can play a key role in supporting adaptation planning and action beyond an individual project. Further, Reed et al (2021) note the complexity of measuring and attributing policy impacts of research and point to the need to consider both positive and negative effects as well as tailoring the impact evaluating design with "aims and context of the evaluation. "…”

Section: Decision-making Contexts and Non-tangible Outcomes Mattermentioning

confidence: 99%

Assessing the Quality of Knowledge for Adaptation–Experiences From Co-designing Climate Services in Sweden

et al. 2021

View full text Add to dashboard Cite

Adaptation to climate change is becoming more urgent, but the wealth of knowledge that informs adaptation planning and decision-making is not used to its full potential. Top-down approaches to knowledge production are identified as one important reason for the gap between science and practice and are criticized for not meeting the needs of intended users. In response to this challenge, there is a growing interest in the creation of user-oriented and actionable climate services to support adaptation. At the same time, recent research suggests that greater efforts are needed to evaluate the effectiveness of knowledge co-production processes and the best criteria by which to gauge the quality of knowledge outcomes, while also considering different stakeholder perspectives. This paper explores these issues through a critical assessment of the quality of knowledge for adaptation generated from a climate services co-design process in two case studies in Sweden. The study draws on experiences from a 5-year research collaboration in which natural and social science researchers, together with local stakeholders, co-designed climate services to support climate adaptation planning and decision-making. The well-established knowledge quality criteria of credibility, legitimacy, saliency, usability, and usefulness remain relevant, but are not sufficient to capture factors relating to whether and how the knowledge actually is applied by climate change adaptation planners and decision-makers. We observe that case-specific circumstances beyond the scope of the co-design process, including the decision-making context as well as non-tangible outcomes, also play crucial roles that should be accounted for in the knowledge assessment processes.

show abstract

“…Given the importance of building nonacademic social capital (see the section on Community), training in influencing strategies, workshop facilitation, stakeholder analysis, and communications are core skills for researchers who want to generate impact from their work. Training in impact planning tools, such as logic models (Rush and Ogborne, 1991;Julian, 1997;see Reed et al, 2018 for an example of a research impact logic model) and Theory of Change (Quinn, 1988), impact monitoring and evaluation methods (e.g., Jancey et al, 2020;Louder et al, 2021;Reed et al, 2021a), and an understanding of the various settings within which impact may be generated, for example skills and strategies for working with policy vs. industry, is also important (e.g., Reed et al, 2018).…”

Section: Skillsmentioning

confidence: 99%

“…Learning capacity is sometimes overlooked in institutional capacity building for impact. Monitoring and evaluation of impacts are important to facilitate learning from mistakes as well as providing evidence to support case studies of impacts that have been successfully achieved (Louder et al, 2021;Reed et al, 2021a). Universities are increasingly investing in impact tracking systems, whether as add-ons to existing research management systems or more sophisticated systems developed specifically for tracking impact, like Vertigo Ventures' Impact Tracker and ResearchFish (Fedorciow and Bayley, 2014;Hill, 2016;McKenna, 2021).…”

Section: Learning Capacitymentioning

confidence: 99%

Impact Culture: Transforming How Universities Tackle Twenty First Century Challenges

Reed

Fazey

2021

Front. Sustain.

View full text Add to dashboard Cite

New ways of doing research are needed to tackle the deep interconnected nature of twenty first century challenges, like climate change, obesity, and entrenched social and economic inequalities. While the impact agenda has been shaping research culture, this has largely been driven by economic imperatives, leading to a range of negative unintended consequences. Alternative approaches are needed to engage researchers in the pursuit of global challenges, but little is known about the role of impact in research cultures, how more or less healthy “impact cultures” might be characterized, or the factors that shape these cultures. We therefore develop a definition, conceptual framework, and typology to explain how different types of impact culture develop and how these cultures may be transformed to empower researchers to co-produce research and action that can tackle societal challenges with relevant stakeholders and publics. A new way of thinking about impact culture is needed to support more societally relevant research. We propose that healthy impact cultures are: (i) based on rigorous, ethical, and action-oriented research; (ii) underpinned by the individual and shared purpose, identities, and values of researchers who create meaning together as they generate impact from their work; (iii) facilitate multiple impact sub-cultures to develop among complementary communities of researchers and stakeholders, which are porous and dynamic, enabling these communities to work together where their needs and interests intersect, as they build trust and connection and attend to the role of social norms and power; and (iv) enabled with sufficient capacity, including skills, resources, leadership, strategic, and learning capacity. Based on this framework we identify four types of culture: corporate impact culture; research “and impact” culture; individualistic impact culture; and co-productive impact culture. We conclude by arguing for a bottom-up transformation of research culture, moving away from the top-down strategies and plans of corporate impact cultures, toward change driven by researchers and stakeholders themselves in more co-productive and participatory impact cultures that can address twenty first century challenges.

show abstract

Evaluating impact from research: A methodological framework

Cited by 137 publications

References 80 publications

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

Assessing the Quality of Knowledge for Adaptation–Experiences From Co-designing Climate Services in Sweden

Impact Culture: Transforming How Universities Tackle Twenty First Century Challenges

Contact Info

Product

Resources

About