The Growth of the Evaluation Tree in the Policy Analysis Forest: Recent Developments in Evaluation

Abstract

The practice and profession of evaluation is continually evolving. From its early origin in the Great Society years of the 1960s, through its golden years of the 1970s, its transformation under the fiscal conservatism of the Reagan era in the 1980s, and in its maturation during the performance and results era of the 1990s, the field of evaluation continues to evolve in response to broader trends in society. This article examines recent developments and trends in the practice and profession of evaluation. Structured around the evaluation theory tree, the presentation of these developments elaborates on the three main branches of evaluation: methods, use, and valuing. The concluding discussion briefly addresses the central role of evaluation-and other types of knowledge production-in providing actionable evidence for use in public policy and program decision making. S48Policy Studies Journal, 48:S1 readers know, policy analysis falls within the political science subfield of policy studies, which addresses both the policy process and analysis activities within that process. As Weimer and Vining (2017) define it, "policy analysis is client-oriented advice relevant to public decisions informed by social values" (p. 30). Broader than evaluation, policy analysis includes almost any analysis that aims to inform policy decisions. Evaluation is a particular type of policy analysis that uses systematic data collection and analysis to determine the worth-often the formative or summative effectiveness-of a program or policy. Formative evaluation considers program processes, and summative evaluation examines a program or policy's ultimate outcomes and impacts. The overlapping but distinct fields of evaluation and policy analysis make the exercise of mapping the evolution and current state of evaluation topical and relevant to program evaluators and policy analysts alike.The article proceeds in three parts. The first part briefly describes the historical development of evaluation in relation to broader social, economic, political, and technological developments. Informed by these observations, the second part focuses on what we see as the most salient recent developments in the practice and profession of evaluation. With the aim of providing a coherent presentation of these diverse trends, we use the "evaluation theory tree" (Christie & Alkin, 2013) as an organizing framework, categorizing the identified trends according to the tree's three main branches: methods, use, and valuing. Part three discusses implications of our observed trends for policy analysis.Before proceeding to this discussion, we note that the methodological foundation of the article is primarily anecdotal, reflecting the combined experience, scholarship, and curiosity of the authors as well as input from experienced colleagues. Collectively, as applied evaluation practitioners and scholars, we have over 60 years as active and committed members of the policy analysis and evaluation communities. We have designed and implemented evaluations across varied policy ...

Does Program Evaluation Affect Program Termination? Insights from the Repeal of Corporate Tax Incentives for the Motion Picture Industry

“…According to the Pew Charitable Trusts (2017), just 10 states "rigorously measure the economic and fiscal impact" of their economic development programs. That comes despite rising demand for program evaluation (Lemire et al, 2020). Therefore, policymakers should expand their analytic capacity, which goes beyond addressing human resource constraints (Bourgeois & Cousins, 2013).…”

“…The role of program evaluation, defined as "analysis that uses systematic data collection and analysis to determine the worth-often the formative or summative effectiveness-of a program" (Lemire, Peck, & Porowski, 2020), in the termination process is also unclear. Indeed, programs with negative evaluations often continue, and those with positive evaluations often do not (Gormley, 2007).…”

Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

“…In designing the key evaluation components of the innovation platform, the evaluation working group drew on Crotty’s [ 27 ] four elements of research design and Lemire and colleagues’ [ 28 ] “evaluation tree”, modified from Christie and Alkin’s [ 29 ] “evaluation theory tree”. These components are outlined in Fig.…”

The 2020 Public Policy Yearbook

“…Dr. Sebastian Lemire, Dr. Laura R. Peck, and Allan Porowski () examine recent developments and trends within the policy evaluation field using the evaluation theory tree as the framework. They discuss how evaluation, as both a practice and a profession, is embedded within the policy enterprise by breaking down the current trends in research as they apply to the three branches of the evaluation tree: methods, use, and valuing.…”