Using automatic speech recognition to assess spoken responses to cognitive tests of semantic verbal fluency

Pakhomov, Serguei; Marino, Susan E.; Banks, Sarah J.; Bernick, Charles

doi:10.1016/j.specom.2015.09.010

Cited by 26 publications

(26 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results show an overall low error rate of 20.01% for the automated system, compared to the manual transcripts. This in itself represents an improvement over results of other authors using ASR systems for evaluating the SVF tasks [53, 54]. In line with previous research, diagnostic groups differ significantly in the number of errors made by ASR (Kruskal-Wallis, χ 2 = 13.7, df = 2, p < 0.001).…”

Section: Discussionsupporting

confidence: 86%

Fully Automatic Speech-Based Analysis of the Semantic Verbal Fluency Task

König

Linz

Tröger

et al. 2018

Dement Geriatr Cogn Disord

View full text Add to dashboard Cite

Background: Semantic verbal fluency (SVF) tests are routinely used in screening for mild cognitive impairment (MCI). In this task, participants name as many items as possible of a semantic category under a time constraint. Clinicians measure task performance manually by summing the number of correct words and errors. More fine-grained variables add valuable information to clinical assessment, but are time-consuming. Therefore, the aim of this study is to investigate whether automatic analysis of the SVF could provide these as accurate as manual and thus, support qualitative screening of neurocognitive impairment. Methods: SVF data were collected from 95 older people with MCI (n = 47), Alzheimer’s or related dementias (ADRD; n = 24), and healthy controls (HC; n = 24). All data were annotated manually and automatically with clusters and switches. The obtained metrics were validated using a classifier to distinguish HC, MCI, and ADRD. Results: Automatically extracted clusters and switches were highly correlated (r = 0.9) with manually established values, and performed as well on the classification task separating HC from persons with ADRD (area under curve [AUC] = 0.939) and MCI (AUC = 0.758). Conclusion: The results show that it is possible to automate fine-grained analyses of SVF data for the assessment of cognitive decline.

show abstract

Section: Discussionsupporting

confidence: 86%

Fully Automatic Speech-Based Analysis of the Semantic Verbal Fluency Task

König

Linz

Tröger

et al. 2018

Dement Geriatr Cogn Disord

View full text Add to dashboard Cite

show abstract

“…Others have used ASR techniques to examine the VF test. In Pakhomov et al (2015), the same Kaldi ASR toolkit (Povey et al, 2011) and in König et al (2018), Google's Automatic Speech Recognition (ASR) service were used for automatic transcription of responses. These studies either attempt to predict the raw VF score based on automatically generated response (Pakhomov et al, 2015) or only investigate count-based measures beside the raw VF score for differentiating MCI from cognitively intact participants (König et al, 2018).…”

Section: Discussionmentioning

confidence: 99%

Improving the Assessment of Mild Cognitive Impairment in Advanced Age With a Novel Multi-Feature Automated Speech and Language Analysis of Verbal Fluency

et al. 2020

View full text Add to dashboard Cite

Introduction: Clinically relevant information can go uncaptured in the conventional scoring of a verbal fluency test. We hypothesize that characterizing the temporal aspects of the response through a set of time related measures will be useful in distinguishing those with MCI from cognitively intact controls. Methods: Audio recordings of an animal fluency test administered to 70 demographically matched older adults (mean age 90.4 years), 28 with mild cognitive impairment (MCI) and 42 cognitively intact (CI) were professionally transcribed and fed into an automatic speech recognition (ASR) system to estimate the start time of each recalled word in the response. Next, we semantically cluster participant generated animal names and through a novel set of time-based measures, we characterize the semantic search strategy of subjects in retrieving words from animal name clusters. This set of time-based features along with standard count-based features (e.g., number of correctly retrieved animal names) were then used in a machine learning algorithm trained for distinguishing those with MCI from CI controls. Results: The combination of both count-based and time-based features, automatically derived from the test response, achieved 77% on AUC-ROC of the support vector machine (SVM) classifier, outperforming the model trained only on the raw test score (AUC, 65%), and well above the chance model (AUC, 50%). Conclusion: This approach supports the value of introducing time-based measures to the assessment of verbal fluency in the context of this generative task differentiating subjects with MCI from those with intact cognition.

show abstract

“…These probabilities represent the confidence with which the ASR system selected the word from possible alternatives for a given portion of the speech signal. In previous work (Pakhomov et al, 2015), we have used this feature to develop a reliability filter that relied on the proportion of words produced by the ASR system with low confidence to identify speech samples that are not suitable for fully automated analysis and may require manual transcription. We applied this strategy to the current study sample and found that, for example, setting aside 30% of the samples for manual transcription improves the ICC to 0.93 for the SVF test and 0.84 for the PVF test on the remaining 70% of the samples (data not shown).…”

Section: Resultsmentioning

confidence: 99%

“…Phonemic verbal fluency and semantic verbal fluency tests were administered using a validated computerized tool (Pakhomov et al, 2015). An iPad app was used to provide standardized test instructions (via iPad) and to audio-record the participant’s responses.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Use of an automated mobile application to assess effects of nicotine withdrawal on verbal fluency: A pilot study.

Pakhomov¹,

Teeple²,

Mills³

et al. 2016

Experimental and Clinical Psychopharmacology

View full text Add to dashboard Cite

Mild-to-moderate impairment in frontally mediated functions such as sustained attention, working memory and inhibition have been found to occur during tobacco withdrawal and may present a barrier to successful cessation. These findings have led to studies evaluating cessation treatments that target nicotine withdrawal related cognitive impairment. The instruments currently used to assess cognitive function provide detailed and specific information but have limitations including being time consuming, cumbersome, anxiety provoking and having poor ecological validity. We examined the feasibility of using a mobile computer application to test verbal fluency (VF) as a quick, easy-to-administer and more ecologically valid method of measuring the effects of short-term smoking abstinence on frontally mediated cognitive functions. Thirty participants completed two assessments – one during ad lib smoking and one after overnight abstinence. At each assessment, semantic and phonemic VF tests were administered using a mobile application and nicotine craving and withdrawal symptom severity was assessed. In repeated assessments, performance on both semantic and phonemic VF tests is expected to improve due to practice effects; however, significant improvements were observed only in semantic (p=0.012) but not phonemic (p=0.154) VF. Additionally, the change between assessments in phonemic (but not semantic) score was significantly associated with withdrawal (p=0.006) and craving (p=0.037) severity measured post-abstinence. This study demonstrates that nicotine withdrawal has differential effects on semantic versus phonemic VF suggesting impairments of working memory, attention, and inhibition. These effects were measured using methods easily employed in large groups of participants, potentially with remote test administration and automated scoring.

show abstract

Using automatic speech recognition to assess spoken responses to cognitive tests of semantic verbal fluency

Cited by 26 publications

References 35 publications

Fully Automatic Speech-Based Analysis of the Semantic Verbal Fluency Task

Fully Automatic Speech-Based Analysis of the Semantic Verbal Fluency Task

Improving the Assessment of Mild Cognitive Impairment in Advanced Age With a Novel Multi-Feature Automated Speech and Language Analysis of Verbal Fluency

Use of an automated mobile application to assess effects of nicotine withdrawal on verbal fluency: A pilot study.

Contact Info

Product

Resources

About