Examining the normality assumption of a design-comparable effect size in single-case designs

Chen, Li-Ting; Chen, Yi-Kai; Yang, Tong-Rong; Chiang, Yu-Shan; Hsieh, Cheng-Yu; Cheng, Che; Ding, Qi-Wen; Wu, Po-Ju; Peng, Chao-Ying Joanne

doi:10.3758/s13428-022-02035-8

Cited by 3 publications

(7 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Articles generally included participants with five or more data points per condition and infrequently featured nonconcurrent designs. These issues, combined with the small number of participants featured in typical D-CES compatible SCD articles ( Mdn = 4), potentially impede the use of SCD to provide evidence of intervention effectiveness in WWC reviews (L. T. Chen et al, 2024).…”

Section: Discussionmentioning

confidence: 99%

“…SCD may only be incorporated into an intervention's WWC evidence rating if results can be quantified in a manner consistent with group experimental designs. Much like group design effect sizes (e.g., Cohen's d), the D-CES provides an estimate of the standardized mean difference between two phases by dividing the difference in the population means of each phase by the total measurement variance across and within cases (L. T. Chen et al, 2024). This approach produces a summary effect for an SCD study suitable for integration into meta-analyses including group designs (Shadish et al, 2015).…”

Section: The Design-comparable Effect Sizementioning

confidence: 99%

“…The lack of research demonstrating the concurrence between the D-CES and visual analysis led Kratochwill et al (2023) to describe WWC's emphasis on the D-CES as "premature" (p. 204). In addition to these issues, the ability of D-CES to approximate variance is compromised in studies with small numbers of measurement instances (e.g., <16) and participants (e.g., <6) under certain conditions (L. T. Chen et al, 2024). Kratochwill et al (2021) also criticized WWC's approach to meta-analysis.…”

Section: The Design-comparable Effect Sizementioning

confidence: 99%

“…Data Points per Condition. Due to concerns regarding the influence of measurement occasions on D-CES (L. T. Chen et al, 2024), we indicated whether the minimum number of adjacent treatment comparisons in qualifying designs (e.g., ABAB) included at least five data points per condition-the minimum inclusion criteria for studies included in Odom et al's (2018) evaluation of the D-CES. Articles in which all participants had the appropriate number of data points were coded as "full."…”

Section: Discretionary Codesmentioning

confidence: 99%

“…The number of participants and measurement instances in a study, rather than being determined by the researcher (cf. L. T. Chen et al, 2024) are often related to the limited availability of certain populations (e.g., people with EBD or developmental disabilities) and the demands of applied settings. One reaction to this reality may be the growing acceptance of procedural adaptations with the potential to save time and accommodate small populations, such as nonconcurrent designs (e.g., Slocum et al, 2022).…”

Section: Single-case and Design-comparable Effect Compatibilitymentioning

confidence: 99%

See 4 more Smart Citations

Examining the impact of design-comparable effect size on the analysis of single-case design in special education.

King,

Nylen,

Enders

et al. 2024

School Psychology

View full text Add to dashboard Cite

Initially excluded from many evaluations of education research, single-case designs have recently received wider acceptance within and beyond special education. The growing approval of single-case design has coincided with an increasing departure from convention, such as the visual analysis of results, and the emphasis on effect sizes comparable with those associated with group designs. The use of designcomparable effect sizes by the What Works Clearinghouse has potential implications for the experimental literature in special education, which is largely composed of single-case designs that may not meet the assumptions required for statistical analysis. This study examined the compatibility of single-case design studies appearing in 33 special education journals with the design-comparable effect sizes and related assumptions described by the What Works Clearinghouse. Of the 1,425 randomly selected single-case design articles published from 1999 to 2021, 59.88% did not satisfy assumptions related to design, number of participants, or treatment replications. The rejection rate varied based on journal emphasis, with publications dedicated to students with developmental disabilities losing the largest proportion of articles. A description of the results follows a discussion of the implications for the interpretation of the evidence base. Impact and ImplicationsA survey of a sample of articles in special education identified single-case designs published over a 22-year period. Analyses noted whether articles were compatible with the design-comparable effect size required in reviews of evidence of the What Works Clearinghouse. Results suggest that a large proportion of studies are eliminated for consideration, which can limit the extent to which scientific evidence can guide the selection of interventions for certain populations and influence research conducted in schools and other applied settings.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: The Design-comparable Effect Sizementioning

confidence: 99%

Section: The Design-comparable Effect Sizementioning

confidence: 99%

Section: Discretionary Codesmentioning

confidence: 99%

Section: Single-case and Design-comparable Effect Compatibilitymentioning

confidence: 99%

See 3 more Smart Citations

Examining the impact of design-comparable effect size on the analysis of single-case design in special education.

King,

Nylen,

Enders

et al. 2024

School Psychology

View full text Add to dashboard Cite

show abstract

Between-case standardized mean differences: Flexible methods for single-case designs

Chen

Pustejovsky

Klingbeil

et al. 2023

Journal of School Psychology

View full text Add to dashboard Cite

Implications of What Works Clearinghouse Guidelines on Single-Case Design: An Investigation of Empty Training Phases

Rodgers,

King

2024

Remedial and Special Education

View full text Add to dashboard Cite

The What Works Clearinghouse quality standards provide guidance regarding studies capable of supporting evidence-based practices. Standards concerning single-case designs have been extensively revised to accommodate new evaluation methods, such as the design comparable effect size. These designs often omit data in which children and other participants receive training. Recently, the What Works Clearinghouse suggested studies with such “empty training phases” do not meet minimum standards of evidence. However, evidence regarding the effect of empty training phases on results is limited. This study used a subset of single-case design data from a recent meta-analysis to simulate studies with empty training phases. We calculated design-comparable effect sizes, evaluated differences between simulated and unsimulated data, and conducted random effects meta-analyses. Effects of simulated studies with artificially designed empty training phases were nearly four times as large as effects of the original, unaltered data. Guidance for intervention researchers follows a description of findings.

show abstract

Examining the normality assumption of a design-comparable effect size in single-case designs

Cited by 3 publications

References 85 publications

Examining the impact of design-comparable effect size on the analysis of single-case design in special education.

Examining the impact of design-comparable effect size on the analysis of single-case design in special education.

Between-case standardized mean differences: Flexible methods for single-case designs

Implications of What Works Clearinghouse Guidelines on Single-Case Design: An Investigation of Empty Training Phases

Contact Info

Product

Resources

About