The Strength of the Methodological Warrants for the Findings of Research on Program Evaluation Use

Abstract

Considerable research has been conducted on the use of the findings of program evaluation, but little, if any, attention has been paid to the soundness of the methods of this research. If the methods are not sound or not well described in the research, the strength of the conclusions of the research is unknown. The authors examine the empirical studies reported in five widely cited reviews of the literature on program evaluation use and summarize the types of methods used, the professions for which they were conducted, and the extent to which they met two basic criteria for soundness. It was concluded that, by and large, the body of studies does not adequately address the criteria. With care, the findings of the research literature on the use of evaluations might provide guidance for evaluators, but they lack sufficient scientific credibility for other uses.

Avaliação de programas de intervenção com adolescentes: limites, avanços e perspectivas

“…No terceiro artigo 17 , os autores buscaram, por meio de uma revisão de literatura, examinar aspectos que fundamentam as bases metodológicas das teorias de avaliação e de suas práticas. Os autores encontraram que a maioria dos estudos desenvolvidos na área (38) está centrada nas características das avaliações.…”

“…No terceiro artigo recuperado 17 , os autores ponderam que os fundamentos metodológicos utilizados para as conclusões sobre o uso da avaliação (e seus instrumentos) são menos fortes do que o desejável. Segundo os autores, os estudos existentes não podem fornecer provas definitivas sobre os efeitos do uso, embora forneçam sugestões.…”

Cross-Continental Reflections on Evaluation Practice

“…Memos were useful for reflection on the purpose of the articles and to glean analytical understanding of how the authors utilized the terms (Maxwell, 2013). For example, a memo crafted from an AJE article (Brandon & Singh, 2009) illustrating the methods, use, and valuing category (refer to Table 3) stated:…”

Understanding the Influence of Independent Civil Society Monitoring and Evaluation at the District Level

“…Triangulation of data sources was done as much as possible to reduce possible biases. For example, one of the common pitfalls of research about evaluation use is the overstatement of use by the respondents (Brandon & Singh, 2009). To reduce this bias as much as possible, the idea was to include only those instances of influence that could be confirmed by other interviewees in the table.…”