Statistical reform: Evidence‐based practice, meta‐analyses, and single subject designs

Abstract

Evidence-based practice approaches to interventions has come of age and promises to provide a new standard of excellence for school psychologists. This article describes several definitions of evidence-based practice and the problems associated with traditional statistical analyses that rely on rejection of the null hypothesis for the establishment of invention effectiveness. Metaanalysis as an approach to ascertain EBPs is reviewed along with the inherent difficulties associated with single subject design research such as autocorrelations. Four meta-analytic approaches are reviewed which include Percentage of Nonoverlapping Data points (PND), the Busk and Serlin: Assumption models, ITSACORR, and Hierarchical Linear Modeling (HLM). HLM is offered as the most promising approach for the analysis for single subject designs. Monte Carlo simulations are modeled with varying degrees of autocorrelations, differing numbers of data points, and simulated effects sizes to show that HLM is an acceptable approach for controlling the risk of Type I errors.

A comparison of mean phase difference and generalized least squares for analyzing single-case data

“…The evidence-based movement in the single-case context has made emphasis on the need for quantitative summaries of the results, especially for making them available for meta-analysis (Jenson, Clark, Kircher, & Kristjansson, 2007). The methodological quality scale (Tate et al, 2008) also includes "statistical analysis" as one of the items of methodological quality.…”

Percentage of nonoverlapping corrected data

“…In the particular case of PND and PNCD, the difference in estimates implies that the interpretation benchmarks proposed by Scruggs and Mastropieri (1998) cannot be applied directly to PNCD. On the other hand, there is evidence that PND is conservative, as compared with other procedures for estimating magnitude of effect (Jenson et al, 2007). Therefore, the effect-size estimates provided by PNCD may resemble more the ones obtained by other models.…”

Optimizing behavioral health interventions with single-case designs: from development to dissemination

“…In addition to visual analysis, several regressionbased approaches are available to analyze timeseries data, such as autoregressive models, robust regression, and hierarchical linear modeling (HLM) [46][47][48][49]. A variety of non-parametric statistics are also available [27].…”

“…The d statistic can also be used to compute power based on the number of observations in each condition and the number of cases in an experiment [57]. In addition, advances in effect size estimates has led to several meta-analyses of results from SCDs [48,[58][59][60][61]. Zucker and associates [62] explored Bayesian mixed-model strategy to combining SCDs using, which allowed population-level claims about the merits of different intervention strategies.…”