Evaluation of Active Living Research

The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews

“…“ Better informed commissioning and contracting ” ([40] p. 17); 5 (16%) led to a change in NHS practice, i.e. “ More effective treatment, screening or management for patients ” ([40], p. 16)This was part of a wider analysis, but in all three areas the projects were reactive; particularly difficult to make an impact with Primary and Community Care researchGold & Taylor, 2007 [41]; United States of AmericaAgency for Healthcare Research and Quality – Integrated delivery systems research networkDocumentary review of programme as a whole and individual projects (50); descriptive interviews (85); four case studies, additional interviewsNo explicit framework describedChanges in operations; “ Of the 50 completed projects studied, 30 had an operational effect or use ” [41] (Operational effect or use is a broad term: so the 60% put into our combined impact category)Success factors: responsiveness of project work to delivery system needs, ongoing funding, development of tools that helped users see their operational relevanceGutman et al, 2009 [42]; United States of AmericaRobert Wood Johnson Foundation – Active living researchA retrospective, in-depth, descriptive study utilising multiple methods; quantitative data derived primarily from a web-based survey of grantee investigators (PIs, Co-PIs), of the 74 projects: 68 responses analysed; qualitative data from 88 interviews with KIsThe conceptual model used in the programme “ was used to guide the evaluation ” ([42], p. S23).Aspects of Weiss's model used for analysing policy contributionsGenerally thought to be too early for much policy impact, but 25% of survey, 43% of interviewees reported a policy impact; however, policy impact in survey could be from active living research in general, not just the specific programme, and could include: “ a specific interaction with policymakers (e.g. testifying, meeting with policymakers, policymaker briefings, etc.)…”

“…testifying, meeting with policymakers, policymaker briefings, etc.) or direct evidence of the research findings in a written policy ” ([42], p. S33)Only 16% of grants had been completed prior to the year of the evaluation; some approaches “ worked well, including developing a multifaceted, ongoing, interactive relationship with advocacy and policymaker organizations ” ([42], p. S32); grantees who completed both interviews and surveys generally gave similar responses, but researchers included in the random sample of interviewees gave higher percentage of policy impact than researchers surveyed; questions slightly different in the interviews than in the surveysHailey et al, 1990 [43]; AustraliaNational Health Technology Advisory Panel – HTA reportsLooked at technologies (20) covered by HTA reports from the panel up to end of 1988. Little provided on methods – presumably desk analysis, just states comparing recommendations, assessments and policy activitiesNo framework describedOut of the first 20 technologies covered by HTA reports there had been significant impact in 11 and probable influence in three: 70% in totalTiming crucial factor for impact; at the margin of our inclusion criteria since it is based more on panel recommendations than a programme of research, but took first 20, not a selectionHailey et al, 2000 [44]; CanadaCanadian province (not stated) – HTA brief tech notesInterviews with those requesting the 20 brief HTA notes (i.e.…”

“…Many of the studies relied on self-reported survey data, and some of them acknowledged potential concerns about such data [51]. Nevertheless, approaches such as triangulation can somewhat mitigate these weaknesses and, in at least four examples, data were collected both by surveys and interviews and, in each case, the self-reported survey data did not seem, on average, to over-emphasise the level of impact [6, 36, 42, 45]. A further limitation with surveys is that the response rate was generally between 50% and 75%, with only four studies receiving replies from more than three-quarters of projects: Kwan et al [51], 87%; Oorwijn et al [55], 79%; Soper and Hanney [61], 83%; and Wooding et al [65], 87%.…”

“…Third, in various studies, it was observed that the impact assessment was occurring at a time that was too early for some, or most, of the research to have had time to make an impact [38, 39, 42, 55, 65]. In such cases, the reported level of impact achieved was not only likely to be lower than it would have been in a later assessment, but also it might look comparatively lower than that from other programmes included in the analysis where the assessment took place some years after the research had been completed.…”

Steps Forward: Review and Recommendations for Research on Walkability, Physical Activity and Cardiovascular Health

“…Research on the built environment and health has been stimulated by research funding initiatives including the Active Living Research program from the Robert Wood Johnson Foundation and the Obesity and the Built Environment initiative from the National Institutes of Health, 21,22 among others. Methodological advances in this work have been made possible by the development of Geographic Information Systems (GIS) software and data resources, 23,24 coupled with statistical software to facilitate work with geographically clustered data using generalized estimating equations, 25 multi-level models, 26 or spatial statistics.…”