False Discovery Rates When Engaging in Skill Versus Performance Deficit Analysis for Academic Instructional Planning

Abstract

Brief experimental analysis (BEA) is a well-researched approach to conducting problem analysis, where potential interventions are pilot tested using a single-subject alternating treatment design. However, its brevity may lead to a high frequency of decision-making errors, particularly in situations where one tested condition is rarely optimal for students (i.e., the base rate). The current study explored the accuracy of a specific variant of BEA, skill versus performance deficit analysis (SPA), across different variations of the basic BEA design, score difference thresholds, and reading and math curriculum-based measurements (CBMs). Findings indicate that the ABAB design provides a reasonable control of such error rates when using reading CBM, whereas subtraction CBM required the use of an ABABAB design. Such error rates could not be controlled, regardless of design, when using multiplication CBM. Implications for best practice in the use of BEA are discussed.

Precision of Single-Skill Math CBM Time-Series Data: The Effect of Probe Stratification and Set Size

“…Christ and Vining (2006) mentioned the possibility of using a stratified item arrangement for SSM-CBM probes but suggested that a random item arrangement is likely sufficient due to the narrower content that is represented on such probes. However, others have expressed concerns related to the precision of SSM-CBM time-series data for instructional decision making (Methe et al, 2015; Solomon et al, in press). Stratifying within SSMs could potentially be a feasible way to increase precision on these probes by increasing the consistency of difficulty across their various iterations.…”

Brief Experimental Analysis of Math Interventions: A Synthesis of Evidence

“…These results are consistent with research in reading BEAs, which has also used a 20% criterion to determine if an intervention is more effective than baseline and visual analysis to determine the most effective intervention overall (Daly et al, 1999; Jones & Wickstrom, 2002; Noell et al, 2001). It has been suggested that more liberal decision-making criteria, such as the 50% criterion may be problematic in that it may result in students being over identified as having a performance-only deficit (Soloman, Dawes, Duhon, & Poncy, 2018). Recently, Soloman et al (2018) conducted an empirical analysis to determine which decision-making criterion would be most appropriate within a skill versus performance analysis and found that the 20% criterion is generally the most appropriate.…”

“…It has been suggested that more liberal decision-making criteria, such as the 50% criterion may be problematic in that it may result in students being over identified as having a performance-only deficit (Soloman, Dawes, Duhon, & Poncy, 2018). Recently, Soloman et al (2018) conducted an empirical analysis to determine which decision-making criterion would be most appropriate within a skill versus performance analysis and found that the 20% criterion is generally the most appropriate. The findings of the current synthesis, along with the recent results of Soloman et al (2018), suggest that in future math BEA studies, a 20% criterion may be the most appropriate and can be used to develop consistency in decision-making procedures across math BEA studies.…”