Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation

Abstract

ObjectivesTo investigate the feasibility of using research papers cited in clinical guidelines as a way to track the impact of particular funding streams or sources.SettingIn recent years, medical research funders have made efforts to enhance the understanding of the impact of their funded research and to provide evidence of the ‘value’ of investments in particular areas of research. One of the most challenging areas of research evaluation is around impact on policy and practice. In the UK, the National Institute of Health and Clinical Excellence (NICE) provide clinical guidelines, which bring together current high-quality evidence on the diagnosis and treatment of clinical problems. Research referenced in these guidelines is an indication of its potential to have real impact on health policy and practice.DesignThis study is based on analysis of the authorship and funding attribution of research cited in two NICE clinical guidelines: dementia and chronic obstructive pulmonary disease.ResultsAnalysis identified that around a third of papers cited in the two NICE guidelines had at least one author based in the UK. In both cases, about half of these UK attributed papers contained acknowledgements which allowed the source of funding for the research to be identified. The research cited in these guidelines was found to have been supported by a diverse set of funders from different sectors. The study also investigated the contribution of research groups based in universities, industry and the public sector.ConclusionsThe study found that there is great potential for guidelines to be used as sources of information on the quality of the research used in their development and that it is possible to track the source of the funding of the research. The challenge is in harnessing the relevant information to track this in an efficient way.

What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield

“…With a growing pool of research in this area, attention is moving toward the evolution of altmetric data relating to policy documents, and what policy makers could or should do with those data (Didegah et al, 2014). Funders should have a natural interest in such impact: if research they have supported is referenced as part of the evidence supporting a national and/or international clinical guideline, for example, then it is an indication that this research is likely to be influencing policy (Kryl et al, 2012). Until recently it was almost impossible to discover whether a piece of research had been cited in a policy document without either manually sifting through potentially relevant documents, by finding out through direct communication between the policy makers, or by pure chance.…”

Are wikipedia citations important evidence of the impact of scholarly articles and books?

“…Clinical guidelines : Online clinical guidelines could be useful to assess the societal impact of individual medical studies by demonstrating their worth when deciding on general health care solutions (Kryl, Allen, Dolby, Sherbon, & Viney, ). A study of the cited references of clinical guidelines produced by the National Institute of Health and Clinical Excellence (NICE) in the United Kingdom found that articles cited in guidelines tend to be more highly cited than comparable articles (Thelwall & Maflahi, ).…”

Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators

“…Although a citation in clinical guidelines does not guarantee an impact on health, it demonstrates a peer perception of research utility and can be considered as an intermediate outcome [26]. Research by the Wellcome Trust, the Medical Research Council (MRC), and the NIHR demonstrated the feasibility of employing conventional bibliometric analysis of research papers cited on clinical guidelines to assess the impact of biomedical research on healthcare policy and practice [24,26,133]. …”