Comparing and Contrasting the National Research Council Report On Evaluating Curricular Effectiveness With the What Works Clearinghouse Approach

Abstract

This article summarizes the findings of the National Research Council (NRC) report On Evaluating Curricular Effectiveness and examines the reviews in middle grades mathematics undertaken by the What Works Clearinghouse (WWC). The NRC report reviewed and assessed 147 key evaluations of 13 National Science Foundation–supported K–12 mathematics curricula and six commercially generated curricula. The report found that the evaluations overall were not sufficiently robust to permit confident judgments on individual programs, so it instead focused on how to define effectiveness in conducting future evaluations. Effectiveness was defined as “an integrated judgment based on interpretation of a number of scientifically valid evaluations that combine social values, empirical evidence, and theoretical rationales” ( NRC, 2004 , p. 4). The report introduced a model for curricular evaluation that includes program theory, implementation, and outcome measures, and reviewed three major methodologies found in the literature: content analysis, comparative analysis, and case study. This article then examines the What Works Clearinghouse’s exclusive emphasis on experimental and quasi-experimental designs from the perspective of an author of the NRC report. The two reports agree in recognizing the need for significant improvement in evaluation quality; however, they differ in four areas: standards for individual studies, need for multiple methods, the way to accumulate information across a set of studies, and how to communicate results with the public. This article concludes with a call for focused efforts to address several shared targets needed to make further progress on how to establish curricular effectiveness.

Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis

“…In a recent criticism of the What Works Clearinghouse, Schoenfeld (2006) expressed concern that because most studies of mathematics curricula use standardized tests or state accountability tests focused more on traditional skills than on concepts and problem solving, there is a serious risk of "false negative" errors, which is to say that studies might miss true and meaningful effects on unmeasured outcomes characteristic of innovative curricula (also see Confrey, 2006, for more on this point). This is indeed a serious problem, and there is no solution to it.…”

“…In addition to concerns about methodological limitations, the report maintained that it is not enough to show differences in student outcomes; curricula, the authors argued, should be reviewed for content by math educators and mathematicians to be sure they correspond to current conceptions of what math content should be. None of the studies combined this kind of curriculum review with rigorous evaluation methods, so the NRC chose not to describe the outcomes it found in the 63 evaluations that met its minimum standards (see Confrey, 2006).…”

“…In 2002, the National Research Council (NRC) convened a blue-ribbon panel to review evaluation data on the effectiveness of mathematics curriculum materials, focusing in particular on innovative programs supported by the National Science Foundation (NSF) but also looking at evaluations of non-NSF materials (Confrey, 2006;NRC, 2004). The NRC panel assembled research evaluating elementary and secondary math programs and ultimately agreed on 63 quasi-experimental studies covering all grade levels, kindergarten through 12, that they considered to meet minimum standards of quality.…”

Using International Study Series and Meta-Analytic Research Syntheses to Scope Pedagogical Development Aimed at Improving Student Attitude and Achievement in School Mathematics and Science

“…However, because a primary aim of Standards-based programmes is also to address non-traditional achievement outcomes, they have been designed to maintain rather than improve student performance on traditional measures (Schoenfeld, 2006). As the reviewers themselves note and others have emphasised (Confrey, 2006;Schoenfeld, 2006), evaluation restricted to traditional measures will miss wider effects:…”

Evaluating and Interpreting Research Syntheses in Adult Learning and Literacy

“…The Best Evidence Encyclopedia (2006) standards are similar to the What Works Clearinghouse standards but also make mention of a criteria requiring a minimal duration for treatments. Finally, Confrey (2006) has compared and contrasted the evaluation standards employed by the National Research Council report on the effectiveness of middle school mathematics curricula, which included the use of multiple methods such as content analysis and case studies, with the What Works Clearinghouse standards that focus exclusively on experiments and quasi-experiments.…”