Interrater Reliability for a Two-Interval, Observer-Based Procedure for Measuring Hearing in Young Children

Abstract

Purpose To overcome methodology limitations for studying auditory development in young children, we have recently developed an observer-based procedure that uses a conditioned, play-based, motor response (see Bonino & Leibold, 2017). The purpose of this article was to examine interrater reliability for the method. Method Video recordings of test sessions of 2- to 4-year-old children ( n = 17) were examined. Detection of a 1000-Hz warble tone was measured with the Play Observer-Based, Two-Interval (PlayO2I) method in each of two conditions: for a fixed intensity level (30 dB SPL) or for a variable intensity level signal (0–30 dB SPL). All test sessions were scored independently by three observers (one real-time, two offline). Observer consensus was evaluated with Fleiss' kappa statistic. To determine if summary data were similar across the observers of each test session, the proportion of correct trials (fixed-level condition) or threshold (variable-level condition) were computed. Results The strength of observer consensus was classified as “almost perfect” and “substantial” for the fixed-level and variable-level conditions, respectively. Follow-up analysis of the variable-level data indicated that differences in observer consensus were seen based on the signal level, the type of response behavior provided by the child, and the confidence level of the real-time observer. Resulting summary data were similar across the three observers of each test session: no significant differences for estimates of the proportion of correct trials or threshold. Conclusions Results from this study confirm strong interrater reliability for the method. The PlayO2I method is a powerful tool for measuring detection and discrimination abilities in young children. Supplemental Material https://doi.org/10.23641/asha.12978197

Development of Masked Speech Detection Thresholds in 2- to 15-year-old Children: Speech-Shaped Noise and Two-Talker Speech Maskers

“…Kappa statistics for individual test sessions ranged from 0.62 to 0.94. These kappa statistics are consistent with previous estimates of inter-rater reliability for the PlayO2I method with a tone detection in quiet task (Bonino et al 2020). Thus, results from this subset of participants suggest excellent inter-rater reliability for the present study.…”

“…Data quality metrics (e.g., probe hit rate, observer confidence) were similar for 2- to 4-year-old children and adults. Moreover, estimates of inter-rater reliability in the present study agreed with those reported for a tone detection in quiet task (Bonino et al 2020). Thus, these results suggest similar success for measuring 2- to 4-year-old children’s hearing abilities with the PlayO2I method for complex stimuli as has been previously reported for simple stimuli.…”

“…Observer confidence was also similar across test sessions with 2- to 4-year-olds and adults. Results from previous work with the PlayO2I method indicate that observer confidence is strongly associated with both the accuracy of the child-observer team (Bonino et al 2019) and estimates of inter-rater reliability (Bonino et al 2020). In the present study, we computed observer confidence as the percentage of trials recorded as “confident” for individual test sessions.…”

“…The PlayO2I method is a powerful tool for researchers to measure detection abilities in young children. Using a tone detection task in quiet, previous research indicates that this method is feasible for 2- to 4-year-old children (Bonino & Leibold 2017), performance is stable over a long test session (Bonino et al 2019), and that there is strong inter-rater reliability for the method (Bonino et al 2020). Building upon this work, this study used the method with an adaptive paradigm to collect masked speech detection thresholds across a wide age range of children.…”

“…In a follow-up study, Bonino et al (2019) confirmed that the accuracy of the child-observer teams was high and remained stable over a long test session (100 trials) for a supra-threshold signal. More recently, strong inter-rater reliability for the method was demonstrated by having three observers independently code test sessions (Bonino et al 2020). Thus, the PlayO2I method is a promising tool for measuring hearing in toddlers and preschoolers—combining the psychometric rigor of the observer-based psychoacoustic procedure with the efficiency of a two-interval, adaptive paradigm.…”