Evaluation of public policy advocacy: challenges, principles and BEST-AC case study

Abstract

Purpose – The purpose of this paper is to demonstrate that both the processes and outcomes of advocacy can be evaluated in ways that can help with learning and accountability. The paper reviews the literature on evaluating advocacy, with a particular focus on development work, and describes an example of the systematic evaluation of the Business Environment Strengthening in Tanzania-Advocacy Component business advocacy programme in Tanzania. Design/methodology/approach – The evaluation uses a Scientific Realist methodology to give a disaggregated, contextual analysis of advocacy, asking the typically Scientific Realist question: “What works for whom in what circumstances?” Complementary methods are being applied longitudinally over a five-year period and include stakeholder interviews, business surveys, diagnostic tools and learning seminars. Findings – The paper argues that advocacy evaluation is no more complex or difficult than other aspects of development. Rigorous, cost-effective methods can be developed, so long as clear conceptualisation is carried out as an initial step. Systematic analysis of influencing tactics and capacity building demonstrates the relative skill of the advocacy organisations and allows the funder to see intermediate indicators of progress which are otherwise invisible. Practical implications – Consistent conceptualisation and measurement allow comparison over time, and between different types of projects and organisations. Integrating methods with the operation of campaigns or programmes allows the evaluator to give feedback in real time and minimise the burden on evaluands. Originality/value – The paper is based on original research/evaluation. The field is heavily concentrated on social change. The paper makes a contribution by providing an example of advocacy evaluation in the field of business advocacy and economic development. In addition, the example extends the field of advocacy evaluation by considering the systematic evaluation of a whole programme of individual advocacy projects.

“Advocates Change the World; Evaluation Can Help”: A Literature Review and Key Insights from the Practice of Advocacy Evaluation

“…Th e low level of use of evaluation by NGOs-only about one quarter of US advocacy organizations have reported that they engage in it ( Innovation Network, 2008 )-is not surprising given that many organizations do not have written advocacy plans to begin with ( Coates & David, 2002 ;Mansfi eld, 2010 ), let alone systematic assessments of this aspect of their work ( Almog-Bar & Schmid, 2014 ;Innovation Network, 2008 ). One reason for the low prevalence of advocacy evaluation is that NGOs have limited human and fi nancial resources to devote to it ( Bagnell Stuart, 2007 ;Coff man, 2009 ;Fine, 2007 ;Jones, 2011 ). A second reason is that advocacy evaluation can be nebulous and poses multiple challenges, which are summarized later in this article.…”

“…Th e literature concurs that the foremost reason for NGOs to undertake assessment of their advocacy work is to gain knowledge to improve strategy ( Almog- Bar & Schmid, 2014 ;Bagnell Stuart, 2007 ;Beer & Reed, 2009 ;Borgman-Arboleda & Clark, 2009 ;Coe & Schlangen, 2011 ;Coff man, 2009 ;Forti, 2012 ;Fyall & McGuire, 2015 ;Guthrie, Louie, & Foster, 2005 ;Innovation Network, 2009 ;Jackson, 2014 ;Mulholland, 2010 ). NGOs engaging in advocacy indicate that the greatest benefi t of evaluation is that it assists them in refi ning strategies and determining the next steps in a campaign ( Innovation Network, 2008 ).…”

“…Just as there is concurrence in the literature about the two main purposes of advocacy evaluation, there is an overwhelming consensus that it is methodologically challenging ( Almog-Bar & Schmid, 2014 ;Jackson, 2014 ;Mansfi eld, 2010 ), and it has earned the distinction of being one of the Evaluation Exchange's "hardto-measure" activities ( Weiss, 2007 ). Some authors point out that program and advocacy evaluation have much in common in their process, as they "conduct systematic and data-based inquiries … quantitative or qualitative … [that] aim to provide high-quality information that has signifi cance or value for whom or what they are evaluating" ( Coff man, 2007 , p. 2).…”

“…A multitude of external factors infl uence systems and change initiatives, and the myriad of players, audiences, and interactions can be diffi cult to track and their infl uence challenging to measure ( Alliance for Justice, 2005 ; Almog- Bar & Schmid, 2014 ;Bagnell Stuart, 2007 ;Coe & Schlangen, 2011 ;Guthrie, Louie, & Foster, 2005 ;Leat, 2012 ;Reisman, Gienapp, & Stachowiak, 2007 ;Teles & Schmitt, 2011 ). Against such a backdrop, it is generally impossible to attribute changes in systems, public opinion, or policies to just one organization's actions ( Alliance for Justice, 2005 ;Bagnell Stuart, 2007 ;Guthrie, Louie, & Foster, 2005 ;Innovation Network, 2009 ;Jones, 2011 ;Klugman, 2010 ;Leat, 2012 ;Reisman, Gienapp, & Stachowiak, 2007 ;Teles & Schmitt, 2011 ).…”

“…Advocacy involves quick cycles of changing strategies in response to an unpredictable external environment, especially when windows of opportunity open: "It's the nature of politics that events evolve rapidly and in a nonlinear fashion, so an eff ort that doesn't seem to be working might suddenly bear fruit, or one that seemed to be on track can suddenly lose momentum" ( Bagnell Stuart, 2007 ;Coates & David, 2002 ;Coff man, 2007 ;Guthrie, Louie, & Foster, 2005 ;Teles & Schmitt, 2011 , p. 39). On the other hand, policy change "oft en runs at the speed of molasses" ( Conner Snibbe, 2006 p. 42;Innovation Network, 2009 ;Jones, 2011 ), occurring over a long time frame, and there may even be dormant periods in which advocacy is best suspended until the external context becomes favourable again ( Forti, 2012 ;Teles & Schmitt, 2011 ). Advocacy organizations, their funders, and evaluators must track progress in the short and medium term while accepting that they cannot determine when or if the ultimate goal might be realized ( Alliance for Justice, 2005 ;Ebrahim, 2005 ;Forti, 2012 ;Guthrie, Louie, & Foster, 2005 ;Kimball, 2011 ;Rees, 2001 ;Reisman, Gienapp, & Stachowiak, 2007 ).…”