Reliability, validity, and accuracy of the intervention selection profile—Function: A brief functional assessment tool.

Abstract

The purpose of this study was to evaluate the reliability, validity, and accuracy of scores from the Intervention Selection Profile-Function (ISP-Function): a brief functional assessment tool founded upon Direct Behavior Rating (DBR) methodology. Participants included 34 teacher-student dyads. Using the ISP-Function, teachers rated the extent to which students exhibited disruptive behavior, as well as the frequency with which disruptions were met with four consequences. Ratings were completed across three 10-min sessions, during which a research assistant also collected systematic direct observation (SDO) data regarding the same behavior and consequences. Results indicated adequate temporal reliability (Ն.70) was attained for the adult attention and peer attention targets across the three ratings; in contrast, up to 8 -18 data points would be needed to achieve adequate reliability across the remaining targets. Findings further suggested that while ISP-Function ratings of disruptive behavior, adult attention, and peer attention were moderately to highly correlated with SDO data, correlations were in the low range for the access to items/activities and escape/avoidance targets. Finally, analysis of difference scores showed that on average, mean ISP-Function scores fell within only 0.33 to 1.81 points of mean SDO scores (on the 0 -10 DBR scale). Agreement coefficients indicative of exact score agreement were less consistent, suggesting accuracy ranged from poor to substantial. Results are promising, but future research is necessary to support applied ISP-Function use. Impact and ImplicationsStudy results support the continued validation of the Intervention Selection Profile-Function, with findings speaking to the reliability, validity, and accuracy of three of five scales within the tool. The availability of such tools enhances the likelihood of schools collecting functional assessment data to inform the implementation of Tier 2 interventions.

Intervention selection profile-function: An examination of decisional accuracy relative to traditional FBA data.

“…Specifically, when reviewing either report, school psychologists frequently agreed that these two consequences were likely functions of student disruptive behavior. This finding aligns with previous studies, which have documented stronger psychometric support for these ISP-Function targets (e.g., Kilgus et al, 2017, 2019). This finding also aligns with what one might expect of attention-based consequences.…”

“…A subsequent study examined the reliability, validity, and accuracy of in vivo ISP-Function ratings (Kilgus et al, 2019). Participants included 34 elementary school students and their classroom teachers.…”

“…The teacher then rates the five ISP-Function items immediately following the rating period. The ISP-Function can be completed consecutively, allowing for multiple observations in a single day (Kilgus et al, 2019). This means that even in the case where 4-6 observations are completed, data collection can be completed within 1-2 days.…”

“…Multiple studies to date have supported the development and initial validation of the ISP-Function (Kilgus et al, 2017(Kilgus et al, , 2019). An initial study examined the accuracy of ISP-Function ratings completed by participants viewing videos of students within a classroom setting.…”

“…Data reports examined in the present study were generated from a previously published study (Kilgus et al, 2019), which examined the reliability, validity, and accuracy of ISP-Function scores. As noted above, participants in that study included 34 teacher-student dyads from eight elementary (K-5) schools in the Midwest and Southeast.…”