Does Research on Evaluation Matter? Findings From a Survey of American Evaluation Association Members and Prominent Evaluation Theorists and Scholars

Abstract

Research on evaluation theories, methods, and practices has increased considerably in the past decade. Even so, little is known about whether published findings from research on evaluation are read by evaluators and whether such findings influence evaluators' thinking about evaluation or their evaluation practice. To address these questions, and others, a random sample of American Evaluation Association (AEA) members and a purposive sample of prominent evaluation theorists and scholars were surveyed. A majority of AEA members (80.95% + 7.60%) and sampled theorists and scholars (84.21%) regularly read research on evaluation and indicate that research on evaluation has influenced their thinking about evaluation and their evaluation practice (97.00% + 3.38% and 94.00% + 4.79%, for AEA members, and 100% and 100%, for prominent theorists and scholars, respectively).

Material Incentives and Other Potential Factors Associated With Response Rates to Internet Surveys of American Evaluation Association Members: Findings From a Randomized Experiment

“…Demonstrably, RoE matters and if findings from RoE are assimilated into evaluation theories, methods, and practices “…evaluators could potentially produce better quality, more useful, and higher impact evaluations” (Coryn et al, 2016, p. 169). In the absence of findings grounded in high-quality empirical evidence, however, the potential influence of RoE is diminished and not likely warranted (Mark, 2007).…”

A Decade of Research on Evaluation

“…Largely, RoE has been considered an academic, or scholarly, endeavor. However, given that less than one third (30.84% as of 2015) of AEA members work in academia (Coryn et al, 2016), it seems imperative that those working in other sectors (e.g., government, business, nonprofit) should be encouraged to contribute to the RoE knowledge base, either directly or indirectly. Such knowledge about evaluation theory, methods, and practices likely differs from that of mainstream academic evaluators (who are more likely to publish in scholarly journals and more widely disseminate findings from RoE).…”

“…For purposes of this review, Coryn et al's (2016) conceptual and operational definition of RoE was used, where RoE is defined as:…”

“…(p. 70)Despite Henry and Mark’s (2003) cynicism, such investigations have seemingly increased in the last decade (e.g., Azzam, 2011; Brandon, 2015a, 2015b; Brandon & Fukunaga, 2013). These studies include surveys of American Evaluation Association (AEA) members (e.g., Coryn et al, 2016; Fleischer & Christie, 2009; Lewis, Harrison, Ah Sam, & Brandon, 2015; Seidling, 2015; Szanyi, Azzam, & Galen, 2013), descriptions of evaluator education and training (e.g., Christie, Quiñones, & Fierro, 2014; Davies & MacKay, 2014; LaVelle & Donaldson, 2010), RoE methods (e.g., Galport & Galport, 2015; Reichardt, 2011; St. Clair, Cook, & Hallberg, 2014), and systematic reviews of published research (e.g., Brandon & Fukunaga, 2013; Chouinard & Cousins, 2013; Coryn, Noakes, Westine, & Schröter, 2011; Johnson et al, 2009; Miller & Campbell, 2006; Vallin, Philippoff, Pierce, & Brandon, 2015), among many others.…”

Evaluators’ Perspectives on Research on Evaluation

“…Given its important function and the increase in RoE activity, it is prudent to develop empirical knowledge about RoE. However, with some exceptions such as a review of the American Journal of Evaluation (AJE) to identify RoE articles (Vallin, Philippoff, Pierce, & Brandon, Chapter 1 of this issue), a similar review by Coryn et al (), a survey of the effects of published RoE on evaluators' thinking (Coryn et al, ), and a report of the results of an RoE needs assessment survey (Szanyi, Azzam, & Galen, 2013), few studies about RoE have been conducted. More could be known, for example, about evaluators’ knowledge of RoE, their attitudes toward it and beliefs about it, and how much they apply its results in their practice or teaching.…”