What are the barriers and facilitators for third sector organisations (non-profits) to evaluate their services? A systematic review

Abstract

BackgroundThe third sector is becoming a more common provider of social and health services, but little is known about how third sector organisations (TSOs) evaluate their activities. Past research has reported that the third sector is under increasing pressure to evaluate its impact and performance by government and other commissioning bodies. However, in responding to this increased pressure to undertake evaluation, research suggests that many TSOs struggle to evaluate their activities following the principles of evidence-based practice (EBP). Yet, there has been no systematic effort to investigate why the third sector is struggling to provide good quality evidence of its effects.MethodsThis systematic review is reported following the PRISMA guidelines. Ten interdisciplinary databases were searched using a search string developed following best practice and in consultation with an information systems expert. Included studies were primary research of any research design investigating barriers to and facilitators of the evaluation process of TSOs as identified by practitioners. All studies were quality appraised, and the results were synthesised as a thematic summary.ResultsTwenty-four studies were included, which mainly investigated TSOs working within health and social services. The thematic summary identified the main barriers for TSOs to undertake evaluation to be related to the (1) lack of financial resources, (2) lack of technical capability and evaluation literacy and (3) challenges around identifying relevant evaluation systems and outcome indicators. Key facilitating factors involved (1) getting the appropriate support, (2) having an organisational culture that supports evaluation and (3) the motivation to be accountable to stakeholders. These findings were robust to study quality.ConclusionsThis review constitutes the first systematic effort to synthesise existing literature on factors supporting and preventing evaluation by TSOs. The prevalence of factors revolving around the lack of support, resources and clarity on appropriate outcome indicators suggests that many of the identified challenges may be met by applying evidence-based and stakeholder-inclusive strategies to develop shared evaluation requirements. Future efforts should address the application of EBP as part of the commissioning process of TSOs.Electronic supplementary materialThe online version of this article (10.1186/s13643-018-0681-1) contains supplementary material, which is available to authorized users.

Reporting and synthesis practices in barriers and facilitators reviews: a methodological systematic review

“…We en up with a broad range of results which were difficult to quantify, count or summarise." [41] Restrictiveness of a barriers and facilitators approach [17,24] "…the synthesis of findings utilised a facilitators and barriers framework, which could mean other factors were excluded. Howe to minimise this risk the search terms did not include 'facilitators barriers' and issues were extracted irrespective of the specific terminology used."…”

“…More specifically, barriers-limite finances, limited education and aware-ness, and location of serv -were also found to be interlinked." [52] [17,18,34,42,51,[57][58][59] "Some articles presented findings related to multiple barriers and perspectives, and these were categorized under multiple themes categories, while not necessarily mutually exclusive [emphasis added], provide an analytical framework from which to consider immigrant women's barriers to cervical cancer screening." [57] "Some facilitators and inhibitors were not exclusive to one theme example, continuity of older people's values, beliefs and persona identity, and factors related to organisational culture."…”

“…Several reviews reported that the struggle to synthesise included research was driven by the fact that many included studies did not explicitly investigate barriers and facilitators, which compelled reviewers to indirectly infer the presence of barriers and facilitators from research not designed to investigate these factors. Although it could be argued that this challenge is grounded in overly broad inclusion criteria, several studies considered such inclusiveness a strength, emphasising the added value of including research that did not explicitly investigate barriers and facilitators [17,24]. Conversely, several studies reported the lack of explicit focus on barriers and facilitators within the primary literature as a limitation to their synthesis [53][54][55].…”

“…A high degree of agreement was achieved between the two reviewers. Two of the included studies were authored by ABM [17,18] and the data extraction and quality assessment of those SRs were therefore done by BV.…”

“…Although there are no established guidelines on how to investigate the reliability of identified factors relating to barriers and facilitators, two studies applied the method suggested by Rees et al [36], in which the robustness of identified themes is tested by, for example, removing studies of poor quality or by investigating whether certain types of research (e.g. qualitative and quantitative) are overrepresented in certain thematic categories versus others [17,18]. Last, only 37 studies discussed their barriers and facilitators approach (discussed in greater detail below).…”