Interobserver agreement: A preliminary investigation into how much is enough?

Abstract

Interobserver agreement (IOA) is important for research and practice, and supports the consistency of behavioral data (Kahng et al., 2011). Although general parameters for how much IOA is needed have been suggested (Bailey & Burch, 2018), it is unknown if the total number of sessions with IOA might impact the IOA coefficient. In this study, IOA was reanalyzed using functional analysis data at various cutoffs. Obtained IOA from these analyses was then compared to the original IOA. Overall, results suggested that, at least when using highly trained observers in a structured clinical setting, there were no significant differences in IOA across cutoffs. However, IOA was sensitive to overall rate of responding in the functional analysis. These data are encouraging, particularly for practitioners, because they provide preliminary support that the amount of sessions with IOA may not be as important as the consistency of the data.

Interobserver agreement and procedural fidelity: An odd asymmetry

“…Most studies we examined reported IOA for measures of the participants' behavior (i.e., dependent variables), which is consistent with prior findings (e.g., Falakfarsa et al, 2022 ; Hausman et al, 2022 ; Kostewicz et al, 2016 ). Far fewer reported IOA for procedural fidelity.…”

“…IOA is now widely reported in research articles published in applied behavior analysis journals and discussed as part of the training of behavior analysts. For example, Hausman et al ( 2022 ) found that 98.9% of studies published in the Journal of Applied Behavior Analysis ( JABA ) from 2014 through 2018 reported IOA. Cooper et al ( 2020 ) devoted 10 pages (pp.…”

On the generality of preference for contingent reinforcement

“…Interrater agreement was measured by having a second, independent rater code 20% of the articles (Hausman et al, 2021; Kratochwill et al, 2010) meeting the inclusion criteria for the variables described above; these articles were selected at random. An agreement was scored if both raters coded an item identically (e.g., both coded “concurrent‐chains” as the method for determining preference).…”

Validity and Reliability Evidence for Assessments Based in Applied Behavior Analysis: A Systematic Review

“…If, for instance, two observers randomly coded 15% of individuals’ performance as “Pass” and 85% of individuals’ performance as “Fail,” their percentage agreement would be about 75% due to chance alone. Furthermore, percent agreement is also influenced by the amount and types of scores collected for calculating agreement (Hausman et al, 2022). An estimate that accounts for chance agreement, such as kappa or weighted kappa, may be more accurate.…”