The Threshold and Inclusive Approaches to Determining “Best Available Evidence”

Abstract

Most evaluators have embraced the goal of evidence-based practice (EBP). Yet, many have criticized EBP review systems that prioritize randomized control trials and use various criteria to limit the studies examined. They suggest this could produce policy recommendations based on small, unrepresentative segments of the literature and recommend a more traditional, inclusive approach. This article reports two empirical studies assessing this criticism, focusing on the What Works Clearinghouse (WWC). An examination of outcomes of 252 WWC reports on literacy interventions found that 6% or fewer of the available studies were selected for review. Half of all intervention reports were based on only one study of a program. Data from 131 studies of a reading curriculum were used to compare conclusions using WWC procedures and more inclusive procedures. Effect estimates from the inclusive approach were more precise and closer to those of other reviews. Implications are discussed.

Current Perspectives on Interventions for Self-Injurious Behavior of Children with Autism Spectrum Disorder: A Systematic Review of the Literature

“…We decided to use an inclusive approach rather than a threshold approach to (a) increase the likelihood of conducting a comprehensive analysis based on the entire body of literature on this topic, as opposed to reviewing a limited sample of studies that met pre-established criteria, and (b) enhance the validity and usefulness of the conclusions drawn. An inclusive approach consists of locating all studies on a topic of interest and then analyzing the quality of the research as a whole, whereas a threshold approach consists of first using a set of pre-established criteria to evaluate the methodological quality of the published studies on a topic of interest and then analyzing only those studies that meet specific criteria (Stockard and Wood 2016).…”

“…We did not use more recent criteria, such as the What Works Clearinghouse (WWC) Standards (Kratochwill et al 2013), to evaluate the methodological quality of the SCRD studies included in the review because these criteria are extremely rigorous, and thus, only a very limited number of studies would have been included in the review. Consequently, conclusions based on a small number of studies that may or may not be representative of the entire body of literature on a specific topic may be misleading (Stockard and Wood 2016).…”

Evidence-Based Practice in the social sciences? A scale of causality, interventions, and possibilities for scientific proof

“…This would help get rid of the tendency to only finance interventions which are proven efficacious or effective via RCTs. Stockard and Wood (2016) investigated how many studies into the effects of educational programs and interventions, performed between 2008 and mid-2014, were included in reports of the What Works Clearinghouse (WWC). The WWC is a United States government institution that examines “what works” in education.…”

“…In 2003 it set high criteria for inclusion of studies, including a focus on RCTs. Stockard and Wood (2016) report that less than two-fifths of the intervention studies performed were included in the reports of the WWC. In a second study, they looked at 131 studies of a reading program.…”

The discordance between evidence and health policy in the United States: the science of translational research and the critical role of diverse stakeholders

“…Since 2007, WWC’s systematic reviews methodology has been revised by Mathematica Policy Research, a private company, but even with those updates their methods fall short. Among other critiques of WWC’s methodology and Mathematica Policy Research’s errors in assessment [ 64 – 66 ], they received particularly strong criticism in two reports from an organisation called the National Institute for Direct Instruction [ 67 , 68 ]. The latter critiques suggest that major concerns in WWC systematic reviews, including “ misinterpretation of study findings, inclusion of studies where programs were not fully implemented, exclusion of relevant studies from review, inappropriate inclusion of studies, concerns over WWC policies and procedures, incorrect information about a program developer and/or publisher, and the classification of programs ” [ 67 ].…”