The Role of Program Theory in Evaluation Research

Abstract

We examined the extent to which mathematics program evaluations that have been conducted according to methodologically rigorous standards have attended to the theories underlying the programs being evaluated. Our analysis focused on the 37 reports of K–12 mathematics program evaluations in the last two decades that have met standards for inclusion in What Works Clearinghouse syntheses. Each report was coded for the extent to which program theory was articulated and used in the evaluation, and for evaluator, study, and program characteristics. Our analyses revealed that very few evaluations attended to program theory, but that aspects of program theory use—as well as reported effect sizes—varied by study and evaluator characteristics.

Theory-based Evaluation of Lesson Study Professional Development: Challenges, Opportunities, and Lessons Learned

“…At the same time, evaluation theorists have long noted enduring challenges in the study and advancement of theory-based evaluation (e.g., Munter et al, 2016). As articulated by Weiss (1997a), these include: challenges associated with the description and use of theory in evaluation practice; challenges associated with selecting appropriate theories that accurately-and fully-underlie program assumptions and processes; difficulty identifying and developing program theory; demands placed on the development and use of methods of data collection and analysis; and complexity and time-intensiveness associated with testing and refining program theories.…”

“…Prior reviews of published program evaluations have noted that evaluations are often not strongly anchored in theory-driven approaches (e.g., Lipsey et al, 1985; Lipsey, 1993). Further, more recent reviews of the role of program theory in guiding evaluation research in STEM education programs noted limited attention of evaluators to program theory (Hamza et al, 2020), finding that the practice of “articulating program theory and using it to inform all phases of an evaluation is yet to become the norm despite repeated calls to do so for almost 30 years” (Munter et al, 2016, p. 23; Rogers, 2007). Compounding this concern is the finding that many evaluations indicating a grounding in program theory continue to rely on and evidence a focus on implementation theory (White, 2009)—that is, evaluations that focus on inputs, activities, outputs, outcomes, and impacts, but “do not examine the causal mechanisms involved in programs and policies […] and do not show the different complementary or alternate causal strands involved in achieving the outcomes” (p. 64).…”

“…These nuances are not unique to educational evaluation (Supovitz & Taylor, 2005). Yet, in many ways, incorporation of theory as a means for developing and guiding program-based evaluations remains in development (Munter et al, 2016) and evaluators are experiencing an increased need to assess the efficacy of instructional programs that are integrated, complex, and multidimensional (DeBarger et al, 2016).…”

Impact and Design of a National-scale Professional Development Program for Mathematics Teachers

“…Researchers have debated which instructional practices are the most effective for improving student achievement (see for example Hmelo-Silver et al, 2007;Kirschner et al, 2006). As demonstrated by Munter et al (2016), theories of instruction for mathematics teacher PD programs carried out in practice also differ in several aspects, such as goals for students' learning and how instruction should be organized to support this learning. Taking goals for students' mathematical learning as an example, some PD programs focus on memorization and the smooth execution of procedures in order to arrive at the right answer (i.e., procedural fluency), while others emphasize conceptual understanding of central mathematical ideas.…”