Combining voice and language features improves automated autism detection

Abstract

Variability in expressive and receptive language, difficulty with pragmatic language, and prosodic difficulties are all features of autism spectrum disorder (ASD). Quantifying language and voice characteristics is an important step for measuring outcomes for autistic people, yet clinical measurement is cumbersome and costly. Using natural language processing (NLP) methods and a harmonic model of speech, we analyzed language transcripts and audio recordings to automatically classify individuals as ASD or non‐ASD. One‐hundred fifty‐eight participants (88 ASD, 70 non‐ASD) ages 7 to 17 were evaluated with the autism diagnostic observation schedule (ADOS‐2), module 3. The ADOS‐2 was transcribed following modified SALT guidelines. Seven automated language measures (ALMs) and 10 automated voice measures (AVMs) for each participant were generated from the transcripts and audio of one ADOS‐2 task. The measures were analyzed using support vector machine (SVM; a binary classifier) and receiver operating characteristic (ROC). The AVM model resulted in an ROC area under the curve (AUC) of 0.7800, the ALM model an AUC of 0.8748, and the combined model a significantly improved AUC of 0.9205. The ALM model better detected ASD participants who were younger and had lower language skills and shorter activity time. ASD participants detected by the AVM model had better language profiles than those detected by the language model. In combination, automated measurement of language and voice characteristics successfully differentiated children with and without autism. This methodology could help design robust outcome measures for future research. Lay Summary People with autism often struggle with communication differences which traditional clinical measures and language tests cannot fully capture. Using language transcripts and audio recordings from 158 children ages 7 to 17, we showed that automated, objective language and voice measurements successfully predict the child's diagnosis. This methodology could help design improved outcome measures for research.

Consistency and reliability of automated language measures across expressive language samples in autism

“…In previous work repetition proportion has consistently underperformed compared to other ALMs (MacFarlane et al, 2022;Salem et al, 2021); here this measure continues to be less consistent and reliable than any other. Our finding that unintelligible proportion is robust to the effect of sampling context could mean that it is an independent measure-a feature not altered by the individual in response to different language activities.…”

“…Interestingly, the ALMs which were more able to discriminate between ASD and non-ASD in previous work-um ratio (a slightly different but comparable measure of disfluency, as discussed in Methods), unintelligible proportion, and CPM, (MacFarlane et al, 2022;Salem et al, 2021)-are also the most consistent across sampling contexts. This result should be further explored by introducing a non-ASD comparison group to future context analysis studies.…”

“…This points to the importance of explicitly defining the language sampling context used in a study of such outcome measures, as different measures may better elicit different aspects of language ability. In prior research (MacFarlane et al, 2022), we found that the prediction of ASD status using voice and language measures was significantly affected by task indices such as length of task and how many words and utterances were spoken, with more accurate predictions occurring for shorter samples. A similar influence could be partially contributing to the effect of conversational context on the ALM results we see here, since different contexts have differing activity length and differing levels of participant talkativeness.…”

Automated speech- and text-based classification of neuropsychiatric conditions in a multidiagnostic setting

“…Classifiers relying on these humandefined features have often reported promising performance in discriminating diagnosed individuals (e.g., with schizophrenia) from controls without any neuropsychiatric condition, with accuracies between 60% and 90% (see Parola, Simonsen, Bliksted, and Fusaroli [3], Koops, Brederoo, Boer, Nadema, Voppel, and Sommer [5], Fusaroli, Lambrechts, Bang, Bowler, and Gaigg [6], and Rybner, Jessen, Mortensen, et al [11] for an overview of existing studies). Furthermore, several studies have sought to combine the two modalities (speech, text) and found them to contain complementary information for the classification of neuropsychiatric conditions [1,21]. This complementarity may be due to the ability of text models to capture variations in word usage that are not captured by acoustic models, and the ability of acoustic models to identify variations in prosody (e.g.…”

“…Machine learning is increasingly used to identify voice and text-based markers that complement current methods for screening and monitoring neuropsychiatric conditions [1,2]. However, while clinical assessment often requires the identification of one diagnosis amongst many with partially overlapping clinical features, virtually all machine learning studies of voice and text-based markers focus on simpler binary outcomes, i.e., patients with one specific diagnosis versus controls without any neuropsychiatric conditions [3][4][5][6][7].…”