A Preliminary Evaluation of Interrater Reliability and Concurrent Validity of Open-Ended Indirect Assessment

Abstract

Indirect assessments are a commonly used component of functional behavior assessment by behavior analysts in practice who work with individuals with severe behavior disorders. Although used frequently, closed-ended indirect assessments have repeatedly been shown to have low to moderate interrater reliability and poor concurrent validity with functional analysis. Recently, the use of open-ended interviews has become more commonly adopted in applied clinical practice, despite no studies evaluating the psychometric properties of such assessments. In the present study, we evaluated the interrater reliability and concurrent validity of an open-ended functional assessment interview. We compared the results of two open-ended indirect assessments conducted with a common caregiver and subsequently conducted functional analyses in an attempt to validate hypotheses generated from the interviews. Interrater agreement for the open-ended interviews was higher than previous research on closed-ended interviews (75%); however, concurrent validity with functional analysis was relatively poor (50%). We discuss these findings in the context of assessment and treatment for severe behavior disorders, as well as best practice methods during functional behavior assessment.

Concurrent Validity of Functional Analysis, ABC Narrative Recordings, and Open-Ended Functional Assessment Interviews

“…When considering a single primary function, the results of our study are similar to those reported by Saini et al (2019) in that both investigations detected correspondence between the functional analysis and the Open-Ended Functional Assessment Interview in approximately half of cases. However, the inclusion of secondary functions considerably increased the concurrent validity as measured in the current study, which is more consistent with other studies on the open-ended interviews (Beaulieu et al, 2018;Jessel et al, 2018;Santiago et al, 2016;Slaton et al, 2017).…”

“…A recent independent study with four participants with developmental disabilities found that the Open-Ended Functional Assessment Interview had a low concurrent internal validity (i.e., 50%) with the results of subsequent functional analyses (Saini et al, 2019). However, Saini et al (2019) experimentally tested only the behavioral functions hypothesized as a function of the challenging behavior following the analysis of the data in the open-ended interview instead of conducting a standard functional analysis that tests all possible hypotheses. Thus, Saini et al (2019) used a procedure that carried the risk of Type II error about the behavioral function.…”

“…However, Saini et al (2019) experimentally tested only the behavioral functions hypothesized as a function of the challenging behavior following the analysis of the data in the open-ended interview instead of conducting a standard functional analysis that tests all possible hypotheses. Thus, Saini et al (2019) used a procedure that carried the risk of Type II error about the behavioral function.…”

Functional Behavior Assessment Practices Used by Canadian Behavioral Health Practitioners

“…With respect to indirect assessments, research on the convergent validity between indirect assessments and functional analysis suggests that results of indirect assessments are accurate approximately 50% of the time (e.g., Paclawskyj, Matson, Rush, Smalls, & Vollmer, ; Saini, Ubdegrove, Biran, & Duncan, in press). More recent literature encourages practitioners to use indirect measures as a means to inform the design of a functional analysis as opposed to informing the development of interventions (Hanley, Jin, Vanselow, & Hanratty, ).…”

Concurrent validity of Open‐Ended Functional Assessment Interviews with functional analysis

“…A recent independent study with four participants with developmental disabilities found that the Open-Ended Functional Assessment Interview had a low concurrent internal validity (i.e., 50%) with the results of subsequent functional analyses (Saini et al, 2019). However, Saini et al (2019) experimentally tested only the behavioral functions hypothesized as a function of the challenging behavior following the analysis of the data in the open-ended interview instead of conducting a standard functional analysis that tests all possible hypotheses. Thus, Saini et al (2019) used a procedure that carried the risk of Type II error (i.e., false negatives) about behavioral function.…”