Building an evidence base for community health: a review of the quality of program evaluations

Abstract

An assessment of the quality of program evaluations conducted in South Australian community health services investigated how effective evaluation reporting is in producing an evidence base for community health. Evaluation reports were assessed by a team of reviewers. Practitioner workshops allowed an understanding of the uses of evaluation and what promotes or acts as a barrier to undertaking evaluations.Community health services do undertake a good deal of evaluation. However, reports were not generally explicit in dealing with the principles that underpin community health. Few engaged with program theory or rationale. Typically, reports were of short-term projects with uncertain futures so there may seem little point in considering issues of longterm health outcomes and transferability to other settings. The most important issue from our study is the lack of investment in applied health services research of the sort that will be required to produce the evidence for practice that policy makers desire. The current lack of evidence for community health reflects failure of the system to invest in research and evaluation that is adequately resourced and THIS PAPER DESCRIBES a review of the quality of program evaluations conducted in South Australian community health services and considers how effective these evaluation reports are in terms of producing an evidence base for the work of these services. It also reports on the attitudes of services toward evaluation. Community health in this paper describes the state government-funded primary health care sector that provides comprehensive primary health care services not targeted at a particular population group. Services are mainly non-medical and multidisciplinary, with a range of strategies and an emphasis on health promotion and illness prevention.Over the last decade there has been an increasing interest in evidence-based medicine (EBM) and the application of evidence-based principles to other areas of health practice and policy making. For example, a MEDLINE search for evidence-based medicine revealed one citation in 1992 but more than 13 000 in 2004. 1 EBM mainly uses systematic reviews of randomised and other controlled trials to assess and synthesise evidence about the effectiveness of interven- What is known about the topic? Program evaluations are often conducted for community health initiatives, but there has been little information on the quality of the completed evaluations. What does this paper add? This paper presents the results of a review of program evaluations conducted in South Australian community health services. Although there were a large number of evaluations completed, most were internal and did not provide useful information for policy and planning decisions. What are the implications for practitioners?The authors suggest a need for investment in health services research to improve the quality of program evaluations for decision making.

Designs and methods used in published Australian health promotion evaluations 1992–2011

“…In contrast, experimental designs can be difficult to implement, an observation Smith and colleagues make in their review of quantitative methods used in Australian health promotion research. 27 It is possible that evaluations examined in this study may have been inadequately funded and underresourced in terms of evaluation time, staff time, and support, which commonly occurs in health promotion, 13,18,20,28 leading to the selection of a more convenient design. The same reasons may explain why surveys were the most common data collection method.…”

“…10,11 In addition, credible evaluations increase respect for health promotion. 12 However, health promotion interventions often are not well evaluated 13,14 and, in some cases, not evaluated at all. 6,15 Inconsistent health promotion evaluation occurs for a range of reasons.…”

“…18 This results in practitioners having fewer skills and resources to draw on for evaluation, contributing to practitioner frustration. 11 Even though numerous frameworks for health promotion evaluation exist to improve the usefulness of evaluations, 13,15,19 some health promotion practitioners may not be aware of appropriate methods to gather evaluation data. 20 Other issues relate to ideological views on health promotion evaluation and their relationship with perceptions of health promotion effectiveness.…”

Framework and tools for planning and evaluating community participation, collaborative partnerships and equity in health promotion

“…Despite the fundamental importance of these principles, they seldom feature in evaluation reporting. 3,4 A review of 93 evaluations conducted in South Australian (SA) metropolitan community health services found that, although there is considerable…”

Reaching those with the greatest need: how Australian primary health care service managers, practitioners and funders understand and respond to health inequity

“…Equity of access to services and in health outcomes are key goals of Primary Health Care (PHC) (World Health Organization, 1978) A review of PHC service evaluations in South Australia (Jolley et al, 2007) found that although equity was a central concern of PHC services it was seldom explicitly evaluated. Additionally, PHC evaluations largely focus on only single programs rather than the whole of the service (Hurley et al, 2010).…”