Machine learning to analyze single‐case graphs: A comparison to visual inspection

Lanovaz, Marc J.; Hranchuk, Kieva

doi:10.1002/jaba.863

Cited by 23 publications

(51 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the effect, we will use the standardized mean difference (SMD), which represents the number of standard deviations above the baseline mean that the behavior has changed. In single-case research, we typically expect SMDs of 3 or more, but testing lower values is important to examine the robustness of the analyses (Lanovaz & Hranchuk, 2021). Graphs with no added effects (SMD = 0) test for Type I error rate whereas graphs with an effect (SMD > 0) allows us to test for power.…”

Section: Step 1 -Generating a Time Seriesmentioning

confidence: 99%

Monte Carlo Simulations to Develop Guidelines for the Use of Single-Case Designs: A Tutorial

Lanovaz¹

2021

Preprint

View full text Add to dashboard Cite

Despite being a cornerstone of the science of behavior analysis, researchers and practitioners often rely on tradition and consensus-based guidelines, rather than empirical evidence, to make decisions about single-case designs. One approach to develop empirically-based guidelines is to use Monte Carlo simulations for validation, but behavior analysts are not necessarily trained to apply this type of methodology. Therefore, the purpose of our technical article is to walk the reader through conducting Monte Carlo simulations to examine the accuracy, Type I error rate, and power of a visual aid for AB graphs using R Code. Additionally, the tutorial provides code to replicate the procedures with single-case experimental designs as well as with the Python programming language. Overall, a broader adoption of Monte Carlo simulations to validate guidelines should lead to an improvement in how researchers and practitioners use single-case designs.

show abstract

Section: Step 1 -Generating a Time Seriesmentioning

confidence: 99%

Monte Carlo Simulations to Develop Guidelines for the Use of Single-Case Designs: A Tutorial

Lanovaz¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Conceptually, we expected these patterns because false positives are more likely when the trend is in the same direction as the expected behavior change. Moreover, changes in Type I error rate are typically accompanied by corresponding changes in power (Lanovaz & Hranchuk, 2021), which explains the variations in the latter. Prior studies had shown that the number of points in Phase B, but not in Phase A, reduced Type I error rates (e.g., Falligant et al, 2020;Lanovaz et al, 2017) when using the CDC.…”

Section: Discussionmentioning

confidence: 99%

“…In the future, researchers should replicate and extend our study by applying the standard deviation criteria proposed by Barnard-Brak et al (2021b) to our dataset. A second limitation is that we did not include visual inspection, which is often described as a recommended practice in the analysis of single-case graphs (Lanovaz & Hranchuk, 2021;Manolov & Vannest, 2019). Future research should continue examining how visual inspection is affected by response-guided decision-making.…”

Section: Discussionmentioning

confidence: 99%

“…However, multiple other methods exist to analyze single-case graphs. To extend prior research, we selected two methods that have never been used to examine response-guided decision making for single-case designs: the conservative dual-criteria (CDC; Fisher et al, 2003) and machine learning (Hranchuk & Lanovaz., 2021). Both these methods perform well with short data series, which may make them relevant methods to examine the effects of response-guided decisions.…”

Section: Waiting For Baseline Stability In Single-case Designs: Is It Worth the Time And Effort?mentioning

confidence: 99%

“…To identify whether each graph showed a clear change, we applied the two objective methods that have been shown to perform best on baseline with few data points: the CDC method (Fisher et al, 2003) and a support vector classifier developed by Lanovaz and Hranchuk (2021).…”

Section: Analysesmentioning

confidence: 99%

See 2 more Smart Citations

Waiting for Baseline Stability in Single-Case Designs: Is It Worth the Time and Effort?

Lanovaz¹,

Primiani²

2021

Preprint

View full text Add to dashboard Cite

Researchers and practitioners often use single-case designs (SCDs), or n-of-1 trials, to develop and validate novel treatments. Standards and guidelines have been published to provide guidance as to how to implement SCDs, but many of their recommendations are not derived from the research literature. For example, one of these recommendations suggests that researchers and practitioners should wait for baseline stability prior to introducing an independent variable. However, this recommendation is not strongly supported by empirical evidence. To address this issue, we used a Monte Carlo simulation to generate a total of 480,000 AB graphs with fixed, response-guided, and random baseline lengths. Then, our analyses compared the Type I error rate and power produced by two methods of analysis: the conservative dual-criteria method (a structured visual aid) and a support vector classifier (a model derived from machine learning). The conservative dual-criteria method produced more power when using response-guided decision-making (i.e., waiting for stability) with negligeable effects on Type I error rate. In contrast, waiting for stability did not reduce decision-making errors with the support vector classifier. Our findings question the necessity of waiting for baseline stability when using SCDs with machine learning, but the study must be replicated with other designs to support our results.

show abstract

Agreement between visual inspection and objective analysis methods: A replication and extension

Taylor

Lanovaz

2022

J of App Behav Analysis

Self Cite

View full text Add to dashboard Cite

Behavior analysts typically rely on visual inspection of single‐case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual‐criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the topic, we examined agreement between the visual inspection of five raters, the conservative dual‐criteria method, and a machine‐learning algorithm (i.e., the support vector classifier) on 198 AB graphs extracted from clinical data. The results indicated that average agreement between the 3 methods was generally consistent. Mean interrater agreement was 84%, whereas raters agreed with the conservative dual‐criteria method and the support vector classifier on 84% and 85% of graphs, respectively. Our results indicate that both objective methods produce results consistent with visual inspection, which may support their future use.

show abstract

Machine learning to analyze single‐case graphs: A comparison to visual inspection

Cited by 23 publications

References 30 publications

Monte Carlo Simulations to Develop Guidelines for the Use of Single-Case Designs: A Tutorial

Monte Carlo Simulations to Develop Guidelines for the Use of Single-Case Designs: A Tutorial

Waiting for Baseline Stability in Single-Case Designs: Is It Worth the Time and Effort?

Agreement between visual inspection and objective analysis methods: A replication and extension

Contact Info

Product

Resources

About