Transforming assessment of speech in children with cleft palate via online crowdsourcing

Abstract

Speech intelligibility is fundamental to social interactions and a critical surgical outcome in patients with cleft palate. Online crowdsourcing is a burgeoning technology, with potential to mitigate the burden of limited accessibility to speech-language-pathologists (SLPs). This pilot study investigates the concordance of online crowdsourced evaluations of hypernasality with SLP ratings of children with cleft palate. Methods Six audio-phrases each from children with cleft palate were assessed by online crowdsourcing using Amazon Mechanical Turk (MTurk), and compared to SLP's gold-standard hypernasality score on the Pittsburgh Weighted Speech Score (PWSS). Phrases were presented to MTurk crowdsourced lay-raters to assess hypernasality on a Likert scale analogous to the PWSS. The survey included clickable reference audio samples for different levels of hypernasality. Results 1,088 unique online crowdsourced speech ratings were collected on 16 sentences of 3 children with cleft palate aged 4-8 years, with audio averaging 6.5 years follow-up after cleft palate surgery. Patient 1 crowd-mean was 2.62 (SLP rated 2-3); Patient 2 crowd-mean 2.66 (SLP rated 3); and Patient 3 crowd-mean 1.76 (SLP rated 2). Rounded for consistency with PWSS scale, all patients matched SLP ratings. Different sentences had different accuracies compared to the SLP gold standard scores.

Crowdsourcing in Plastic & Reconstructive Surgery: A Systematic Review with Reporting Recommendations

“…Forty-five articles were ultimately included for data extraction (Table 1). 13–57 Proportionate rater agreement was 91.7% for abstract screening and 92.5% for full-text screening. The Cohen kappa coefficient for interrater reliability was 0.82 for abstract screening and 0.81 for full-text screening, demonstrating sufficient interrater reliability.…”

“…We identified seven studies making these comparisons within our own specialty, the concordances between groups being quite variable (Table 5). It is paradoxical that the degree of complexity or sophistication of the topic does not seem to predict the agreement of layperson and experts; for example, crowds were found to agree with experts on ratings of hypernasal speech, 44 but disagree on a five-option comparison of eyelid swelling 43 or the aesthetics of umbilicoplasty. 52 Because the number of studies evaluating these comparisons is quite small, more studies are needed to determine the areas in which crowds may provide expert-level comparable ratings.…”

Seeing Cleft Lip from a New Angle: Crowdsourcing to Determine Whether Scar Severity or Lip Angle Matters More to the General Public

“…Previously, crowdsourcing has been successfully used in medicine, 15,16 including the evaluation of surgical skill, 17 detection of abnormality, 18–20 and assessment of patient perspectives on health care. 21 More recently, crowdsourcing’s applications have spread to plastic surgery and cleft outcomes, with applications in cleft speech, 8 lip appearance, 9 and orthognathic surgery results. 22 This article represents a novel approach in multiple aspects, not only for its direct evaluation of the relative importance of scar severity versus Cupid’s bow peak angle, but in its approach to cleft outcomes.…”

“…Online crowdsourcing is a burgeoning technology that allows researchers to leverage large-scale opinions of the general public. Platforms such as Amazon Mechanical Turk (MTurk), have previously investigated the concordance of layperson crowdsourced worker ratings with ratings of experts in the domains of cleft speech and Asher-McDade fixed-scale ratings, [8][9][10] and used MTurk to compare aesthetic outcomes in cleft lip repair. 11,12 Encouraged by the results of these studies, which found that online crowdsourced ratings are highly consistent with expert opinion, we aim to understand lay perspective of cleft outcomes using a novel technique of simulating different lip angles and scar thicknesses.…”

The Feasibility of Cross-Linguistic Speech Evaluation in the Care of International Cleft Palate Patients

“…Finally, it is important to note that there is burgeoning interest in using new technologies to evaluate CP outcomes, such as virtual crowdsourcing and deep learning algo-rithms. [31][32][33][34] Once established, these strategies could be quickly adopted on a global scale and used in low-resource areas to help alleviate disparities in cleft care.…”