Detecting False Positives in A-B Designs

Krueger, Tyler K.; Rapp, John T.; Ott, Lisa M.; Lood, Elizabeth A.; Novotny, Marissa A.

doi:10.1177/0145445512468754

Cited by 18 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, we did not examine the differential effects of autocorrelation and specific phase lengths on accuracy (e.g., Fisher et al, 2003;Lanovaz et al, 2017). As a next step, researchers should conduct parametric analyses of autocorrelation and phase length while also comparing the results with other structured or statistical aids for the analysis of AB or multiple baseline designs (Ferron, Joo, & Levin, 2017;Giannakakos & Lanovaz, 2019;Krueger, Rapp, Ott, Lood, & Novotny, 2013;Manolov & Vannest, 2019). A final limitation is the large amount of data, or exemplars, required to train machine-learning models.…”

Section: Discussionmentioning

confidence: 99%

Machine Learning to Analyze Single-Case Data: A Proof of Concept

2020

View full text Add to dashboard Cite

Visual analysis is the most commonly used method for interpreting data from singlecase designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Machine Learning to Analyze Single-Case Data: A Proof of Concept

2020

View full text Add to dashboard Cite

show abstract

“…The magnitude of this problem is, however, open to debate. Using Monte Carlo simulations, researchers have shown that the probability of a type 1 error is low (i.e., < 0.05) when AB data are examined via visual analysis employing structured criteria (Fisher, Kelley, & Lomas, 2003;Krueger, Rapp, Ott, Lood, & Novotny, 2013;Novotny et al, 2014). One potential limitation of using simulated data is that the error may not correctly mimic patterns observed with human participants.…”

mentioning

confidence: 99%

Using Single-Case Designs in Practical Settings: Is Within-Subject Replication Always Necessary?

et al. 2018

View full text Add to dashboard Cite

Behavior analysts have widely adopted and embraced within-subject replication through the use of reversal and multielement designs. However, the withdrawal of treatment, which is central to these designs, may not be desirable, feasible, or even ethical in practical settings. To examine this issue, we extracted 501 ABAB graphs from theses and dissertations to examine to what extent we would have reached correct or incorrect conclusions if we had based our analysis on the initial AB component only. In our first experiment, we examined the proportion of datasets for which the results of the first AB component matched the results of the subsequent phase reversals. In our second experiment, we calculated three effect size estimates for the same datasets to examine whether these measures could predict the relevance of conducting a withinsubject replication. Our analyses indicated that the initial effects were successfully replicated at least once in approximately 85% of the cases and that effect size may predict the probability of within-subject replication. Overall, our results support the rather controversial proposition that it may be possible to set threshold values of effect size above which conducting a replication could be considered unnecessary. That said, more research is needed to confirm and examine the generalizability of these results prior to recommending changes in practice.

show abstract

“…Second, our study limited its analysis to the dual-criteria method and visual inspection. In the future, researchers may also explore other methods of data analyses to examine whether the results remain consistent (e.g., Krueger et al, 2013, Manolov & Vannest, 2020. Third, our analyses only involved simulated data as it is impossible to measure power on nonsimulated data without delving into circular reasoning.…”

Section: Discussionmentioning

confidence: 99%

How Many Tiers Do We Need? Type I Errors and Power in Multiple Baseline Designs

Lanovaz¹,

Turgeon²

2020

Preprint

View full text Add to dashboard Cite

Design quality guidelines typically recommend that multiple baseline designs include at least three demonstrations of effects. Despite its widespread adoption, this recommendation does not appear grounded in empirical evidence. The main purpose of our study was to address this issue by assessing Type I error rate and power in multiple baseline designs. First, we generated 10,000 multiple baseline graphs, applied the dual-criteria method to each tier, and computed Type I error rate and power for different number of tiers showing a clear change. Second, two raters categorized the tiers for 300 multiple baseline graphs to replicate our analyses using visual inspection. When multiple baseline designs had at least three tiers and two or more of these tiers showed a clear change, the Type I error rate remained adequate (< .05) while power also reached acceptable levels (> .80). In contrast, requiring all tiers to show a clear change resulted in overly stringent conclusions (i.e., unacceptably low power). Therefore, researchers and practitioners should carefully consider limitations in power when requiring all tiers of a multiple baseline design to show a clear change in their analyses.

show abstract

Detecting False Positives in A-B Designs

Cited by 18 publications

References 20 publications

Machine Learning to Analyze Single-Case Data: A Proof of Concept

Machine Learning to Analyze Single-Case Data: A Proof of Concept

Using Single-Case Designs in Practical Settings: Is Within-Subject Replication Always Necessary?

How Many Tiers Do We Need? Type I Errors and Power in Multiple Baseline Designs

Contact Info

Product

Resources

About