Assessing Observer Accuracy in Continuous Recording of Rate and Duration: Three Algorithms Compared

Abstract

The three algorithms most frequently selected by behavior-analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block-by-block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time-window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).

Reinforcer Variation in a Token Economy

“…Interobserver agreement was calculated by using block-by-block agreement (Mudford, Martin, Hui, & Taylor, 2009). Blocks were divided into trials for the preference assessment and into 10-s bins for the other assessments.…”

Calibration of Observational Measurement of Rate of Responding

“…Continuous measurement of free‐operant responding by observers recording on handheld or laptop computers has become more common in applied behavior analysis research than discontinuous recording (e.g., momentary time sampling or partial‐interval recording; Mudford, Taylor, & Martin, 2009). A benefit for applied behavior analysis, as a natural science of human behavior, is that the products of measurement can be reported in conventional units (e.g., responses per minute) from recording of behavioral events.…”

“…Behavior analysts who present their data in conventional units of measurement have the opportunity to bring their methods for assessing and reporting the quality of those data in line with other natural sciences. We use the overarching term quality to include concepts of accuracy, precision, and errors of measurement from the science of measurement (metrology) (e.g., Hauck, Koch, Abernethy, & Williams, 2008; The Royal Society of Chemistry, 2003); interobserver agreement from applied behavior analysis (e.g., Mudford, Taylor, et al, 2009); and reliability and validity from social sciences (e.g., Gresham, 2003). Metrological concepts and interobserver agreement will be defined and described.…”

“…In other natural sciences, the accuracy of a measurement instrument is typically determined by comparing or calibrating the instrument's measurements with known standards (e.g., testing the accuracy of a balance scale by measuring a set of objects of known weight). Although accuracy has long been recognized as the gold standard for assessing the quality of observational measurement (e.g., Cooper, Heron, & Heward, 1987, 2007; Johnston & Pennypacker, 1980, 1993, 2009), applied behavior analysts have rarely assessed the accuracy of their continuous behavioral measurement systems (Mudford, Martin, Hui, & Taylor, 2009; Mudford, Taylor, et al, 2009).…”

Examination of the Influence of Contingency on Changes in Reinforcer Value

“…Mean percentage agreement across participants for the PR analyses was 99% (range, 92% to 100%). Interobserver agreement data for the reinforcer manipulation sessions were calculated for compliance with prompts, delivery of the stimulus, and item interaction using the exact agreement within intervals method (Mudford, Martin, Hui, & Taylor, 2009). Sessions were divided into consecutive 10‐s intervals.…”