Finding the experts in the crowd: Validity and reliability of crowdsourced measures of children’s gradient speech contrasts

Abstract

Perceptual ratings aggregated across multiple non-expert listeners can be used to measure covert contrast in child speech. Online crowdsourcing provides access to a large pool of raters, but for practical purposes, researchers may wish to use smaller samples. The ratings obtained from these smaller samples may not maintain the high levels of validity seen in larger samples. This study aims to measure the validity and reliability of crowdsourced continuous ratings of child speech, obtained through Visual Analog Scaling, and to identify ways to improve these measurements. We first assess overall validity and interrater reliability for measurements obtained from a large set of raters. Second, we investigate two rater-level measures of quality, individual validity and intrarater reliability, and examine the relationship between them. Third, we show these estimates may be used to establish guidelines for the inclusion of raters, thus impacting the quality of results obtained when smaller samples are used.

Deriving gradient measures of child speech from crowdsourced ratings

“…Validity relative to an external acoustic gold standard was found to be highly correlated with intra-rater reliability across repeated presentations, suggesting that reliability across repeated ratings could represent a useful method to screen raters. However, even when applying a relatively stringent standard and including only the 60 most reliable out of 120 raters, Harel et al (2016) continued to find considerable variability in bootstrapped resamples of n = 9 listeners. Because AMT raters need to be compensated for their work even in the context of a pre-screening test, it was judged impractical to apply an even stricter standard to select only raters with truly expert-like performance.…”

“…If there were an effective method to identify other crowdsourced listeners who exhibit expert- or near-expert-level performance, VAS ratings from as few as three listeners might yield valid information about the gradient properties of speech tokens. Recent work by Harel et al (2016) investigated this question of “finding the experts in the crowd.” The performance of individual AMT raters was assessed by measuring both reliability and validity of VAS ratings obtained when a small set of tokens was presented for repeated rating. Validity relative to an external acoustic gold standard was found to be highly correlated with intra-rater reliability across repeated presentations, suggesting that reliability across repeated ratings could represent a useful method to screen raters.…”

“…An intermediate option would be to collect ratings from a large number of students in speech pathology, who are a conveniently accessible population for most researchers, and whose services can usually be engaged at lower cost than experienced speech-language pathologists. Because students are neither experts nor truly naive, researchers should either recruit large numbers of students or use some form of pre-screening to identify students whose perceptual ratings are accurate and reliable (Harel et al, 2016). Finally, a typical clinical practitioner is unlikely to have the time and resources to obtain and analyze blinded listener ratings using currently available systems.…”

Efficacy of Electropalatography for Treating Misarticulation of /r/

“…Blinded expert ratings were not available for within-treatment productions due to the large number of trials (N = 4,800) elicited in this context. However, Harel, Hitchcock, Szeredi, Ortiz, and McAllister Byun (2016) found that reliability measures estimated from a small subset of data were highly correlated with the same measure as derived from a larger subset of the data from the same individuals. Such findings suggest that one set of data can be a useful predictor of future ratings and support the possibility that the high level of agreement observed to hold between the clinician and the blinded raters on baseline and maintenance probes might also hold for within-session data.…”

The Effect of Formant Biofeedback on the Feminization of Voice in Transgender Women

“…To increase the reliability of these ratings, prior to completing experimental rating blocks, listeners were required to pass an eligibilitytesting block measuring the reliability with which they used the VAS to rate speech tokens. Following a protocol developed in previous research on the use of repeated ratings to identify high-performing individuals among crowdsourced listeners, 39 a sample of 30 tokens were repeated four times in random order, totaling 120 tokens. Raters were excluded if their intraclass correlation coefficient across the repeated ratings was lower than 0.8.…”