Deriving gradient measures of child speech from crowdsourced ratings

Abstract

Recent research has demonstrated that perceptual ratings aggregated across multiple non-expert listeners can reveal gradient degrees of covert contrast between target and error sounds that listeners might transcribe identically. Aggregated ratings have been found to correlate strongly with acoustic gold standard measures both when individual raters use a continuous rating scale such as visual analog scaling (Munson, Johnson, & Edwards, 2012) and when individual raters provide binary ratings (McAllister Byun, Halpin, & Szeredi, 2015). In light of evidence that inexperienced listeners use continuous scales less consistently than experienced listeners, this study investigated the relative merits of binary versus continuous rating scales when aggregating responses over large numbers of naive listeners recruited through online crowdsourcing. Stimuli were words produced by children in treatment for misarticulation of North American English /r/. Each listener rated the same 40 tokens two times: once using Visual Analog Scaling (VAS) and once using a binary rating scale. The gradient rhoticity of each item was then estimated using (a) VAS click location, averaged across raters; (b) the proportion of raters who assigned the “correct /r/” label to each item in the binary rating task (p̂). First, we validate these two measures of rhoticity against each other and against an acoustic gold standard. Second, we explore the range of variability in individual response patterns that underlie these group-level data. Third, we integrate statistical, theoretical, and practical considerations to offer guidelines for determining which measure to use in a given situation.

Differential Effects of Visual-Acoustic Biofeedback Intervention for Residual Speech Errors

“…They found that binary ratings aggregated over 250 naïve listeners on AMT were highly correlated with binary ratings aggregated over 25 expert listeners ( r = 0.92) and with an acoustic measure of rhoticity, F3-F2 distance ( r = -0.79). Bootstrap analyses revealed that when nine or more AMT listeners were included in a subsample of raters, the aggregated AMT ratings converged with ratings aggregated over subsamples of three expert listeners, considered the “industry standard.” The accuracy of each token can be estimated by aggregating responses across raters using correct , which is the proportion of listeners who in a binary forced-choice task responded that the token was a “correct r sound” (McAllister Byun et al, 2016a). …”

Treatment for Residual Rhotic Errors With High- and Low-Frequency Ultrasound Visual Feedback: A Single-Case Experimental Design

“…3 All participants had U.S.-based IP addresses and, per self-report, were native speakers of English with no history of speech or hearing impairment. When aggregating accuracy ratings across listeners, we usep correct , defined as the percentage of "correct" ratings out of all ratings, pooled across listeners (McAllister Byun, Harel, Halpin, & Szeredi, 2016).…”

Protocol for Correcting Residual Errors with Spectral, ULtrasound, Traditional Speech therapy Randomized Controlled Trial (C-RESULTS RCT)

“…(Perceptual ratings are not feasible for Phase 1 because within-treatment trials may be prolonged or otherwise unnatural, posing a confound for accuracy ratings provided by untrained listeners.) Following protocols refined in our previous research [42,43], we will split each word probe recording into 50 word-level productions and pool these productions across speakers and time points. These stimuli will then be presented in a randomized order for online listeners who originate from US-based IP addresses and report speaking American English as their native language.…”

“…Per previous research, ratings will be collected until at least 9 unique listeners have rated each token. The proportion of "correct" ratings out of the total number of ratings (p-correct), which correlates strongly with both expert listeners' ratings and acoustic measures [42,43], will serve as our primary perceptual measure of /ɹ/ production accuracy.…”