Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints

Abstract

ArticlesArticles should deal wth topics applicable to the broad field of program evaluation. Articles may focus on evaluation methods, theory, practice, or findings. In all cases, implications for practicing evaluators should be clearly identified. Examples of contributions include, but are not limited to, reviews of new developments in evaluation, description of a current evaluation study, critical reviews of some area of evaluation practice, and presentations of important new techniques. Manuscripts should follow APA format for references and style. Length per se is not a criterion in evaluating submissions. ABSTRACTThe paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the "Shoestring Evaluation" approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the Michael Bamberger • 6295 S.W. combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced.

An evaluation of the use of ICT within primary education in Malawi

“…It also ensured that data gathered was not solely anecdotal but credible, dependable and confirmable [30]. Within this overall aim for increased creativity and rigor, the research remained subject to the standard complexities and constraints of limited time, budget, data and personnel [31] [32].…”

Measuring training participants’ changing performance using self-reporting methods and their implication in the Grameen Bank training evaluation

“…Bamberger, Church and Fort (2004), Eckert (2001), Haccoun and Hamtiaux (1994), and Mayne (1999) have different training evaluation methods, which do not use self-assessment training evaluation methods. For example, Haccoun and Hamtiaux (1994) use a simple procedure for estimating the effectiveness of training on trainee knowledge through the Internal Referencing Strategy (IRS) as compared to a traditional experimental evaluation (p. 593).…”

“…GB's training evaluation data collection uses the pre-test and post-test control group and experimental group (Eckert method) and uses IRS with pre-post single group training evaluation design (Haccoun & Hamitaux, 1994), which involves many stages. Moreover, the use of Mayne's training performance validity measurement and Bamberger, Church and Fort's (2004) shoestring method incorporating checklists for quantitative secondary data collection will be complicated, expensive, and cumbersome, and grassroots employees may feel uneasy and nervous to go through all the steps of training evaluation. Hence, asking questions of trainees about learning gains following the retrospective method could be one method in the GB training measurement.…”

Évaluer les activités de construction de la paix dans les situations de conflit et de fragilité

“…Une théorie du programme combine souvent une théorie du changement et un modèle de mise en oeuvre (Bamberger et al, 2006).…”