National Evaluation of the Emergency School Aid Act (ESAA): a Review of Methodological Issues

Abstract

Large-scale field evaluations of education programs typically present complex and competing design requirements that can rarely be satisfied by ideal, textbook solutions. This paper uses a recently completed national evaluation of the federally-funded Emergency School Aid Act (ESAA) Program to illustrate in concrete fashion some of the problems often encountered in major program evaluations, and traces the evolution of efforts in that three-year longitudinal study-both in the original design conceptualization and in the actual implementation and data analysis phases-to resolve competing demands and to provide as much methodological rigor as possible under field conditions. Issues discussed here include the selection of experimental versus quasi-experimental designs; the development of sampling procedures to provide maximum precision in treatment-control comparisons; the selection of achievement tests and difficulties in developing and administering other, non-cognitive outcome measures; and the importance of ascertaining whether the underlying assumptions of a true experimental design have been met before conclusions about program impact are drawn on the basis of treatment-control comparisons. at OAKLAND UNIV on June 8, 2015 http://jebs.aera.net Downloaded from 2 Coulson I. at OAKLAND UNIV on June 8, 2015 http://jebs.aera.net Downloaded from ESAA: Methodological Issues 3Readers may note that the report tends to dwell on problems, on difficult and sometimes controversial design decisions, and on compromise approaches. Little attention is paid to the more routine aspects of the evaluation in which traditional, textbook, solutions were applied to the evaluation design, the data collection efforts, or the data analyses. The reason for this approach is that it seems more important to convey a realistic picture of large-scale evaluations, where design tradeoffs, expedient solutions, and dirty data often prevail, than to sing the praise of the ESAA evaluation. Furthermore, there is much truth in the old saw that more is learned from difficulties than from simple solutions. Textbooks and graduate schools teach ideal solutions, but only practical experience can prepare researchers for the difficult decisions required in field studies. To keep things in perspective, however, it is important to point out that the ESAA evaluation was, by any standards, among the more rigorously designed and implemented studies of education programs ever conducted on a national scale. This rigor stemmed not only from the study's design and analyses, which emphasized experimental and quasi-experimental techniques, but also from the care that went into the instrument development efforts, and from the use of trained, independent data collectors for test administration, interviews and in-depth observations. Despite compromises required along the way, the basic integrity of the study's design was sufficiently well preserved so that considerable confidence can be placed in the evaluation findings. n. THE PROGRAMS BEING EVALUATED Although SDC's ESAA...

8: Implications of Educational Evaluation for Evaluation Policy

“…The most recent examples include evaluations of: parts of the Emergency School Aid Act (Coulson, 1978); a subset of career education programs (NIE, 1977;Datta, Note 3); at least one Middle Start program (Yinger, Ikeda, Laycock, & Cutler, 1977); educational TV programs in health (Mielke & Swinehart, 1976;Minor & Bradburn, 1976); preschool education (Bogatz & Ball, 1971); primary education (Ball & Bogatz, 1973); radio-based mathematics instruction (Searle, Friend, & Suppes, 1976); and even grade retention (Jackson, 1975b). Moles (Note 4) has managed to implement randomized tests of programs that were designed to reduce disruptive school behavior.…”

“…Some field experiments appear to have been mounted without prior testing of this sort. However, evaluations such as the System Development Corporation's test of ESAA (Coulson, 1978) are generally pre-ceded by imperfect results in other areas. This earlier experience counts more heavily in the applied sciences than it does in basic research where poor designs are less public and therefore less embarrassing.…”

Politics, Process, and Pyramids

“…Of the many noteworthy features of John Coulson's (1978) review of the Emergency School Aid Act (ESAA) evaluation, the most important may be the perspective that is provided on the many methodological problems that arise in such large-scale evaluation efforts. Technical writings on evaluation have been dominated by proposals, counterproposals, and rebuttals of various statistical adjustment techniques, such as analysis of covariance, which purport to make valid assessments of treatment effects from data on nonequivalent groups.…”

How Much Does ESAA Really Accelerate Academic Growth?

“…There are many interesting aspects to Coulson's (1978) paper, relating not only to the Emergency School Aid Act (ESAA) evaluation itself, but to its implications for broader debates about evaluation methodology. There seems to be at present an increasing skepticism about the usefulness of controlled experiments for educational evaluation, and particularly for evaluation of large-scale, federally funded programs (e.g., Parlett and Hamilton, 1976;Ross and Cronbach, 1976;and Weiss and Rein, 1972).…”