A Linguistically-Informed Fusion Approach for Multimodal Depression Detection

Morales, Michelle; Scherer, Stefan; Levitan, Rivka

doi:10.18653/v1/w18-0602

Cited by 24 publications

(23 citation statements)

References 21 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With increasing loneliness scores, speech responses tended to have less inflections and longer pauses in prosodic features; reduced second formant frequencies and variances of the speech spectrum (ΔMFCCs) in acoustic features; and fewer positive words and more filler words in linguistic features. All these trends in their changes were consistent with those observed in individuals with changes in emotional states and mental health conditions, especially those reported in previous studies on depressed speech [for F2 ( 35 , 36 , 40 ); for the variance of ΔMFCCs ( 49 , 50 ); for pitch variation ( 41 , 50 ); for pauses ( 34 , 38 ); for positive words ( 33 , 37 ); for filler words ( 55 – 57 )]. This result may be reasonable because loneliness and depression are different constructs but closely correlated with each other ( 11 ).…”

Section: Discussionsupporting

confidence: 90%

“…For example, several studies reported that depressed individuals tended to use more negative words and fewer positive words than non-depressed individuals ( 33 , 37 ). Filler words are commonly found in spontaneous speech and have been suggested as important signatures for detecting depression ( 55 – 57 ). We thus used the number of positive and negative words and the proportion of filler words as linguistic features.…”

Section: Methodsmentioning

confidence: 99%

“…From the speech responses of each participant to the eight questions, we automatically extracted a total of 160 speech features consisting of 128 acoustic features, 16 prosodic features, and 16 linguistic features. These features were determined on the basis of previous studies on inferring changes in emotional states and mental health conditions such as depression and suicidality (27,(33)(34)(35)(36)(37)(38)(39)(40)(41)(42)(43)(55)(56)(57). Full list of speech features is available in Supplementary Table 2.…”

Section: Speech Data Analysismentioning

confidence: 99%

See 2 more Smart Citations

Automatic Assessment of Loneliness in Older Adults Using Speech Analysis on Responses to Daily Life Questions

et al. 2021

View full text Add to dashboard Cite

Loneliness is a perceived state of social and emotional isolation that has been associated with a wide range of adverse health effects in older adults. Automatically assessing loneliness by passively monitoring daily behaviors could potentially contribute to early detection and intervention for mitigating loneliness. Speech data has been successfully used for inferring changes in emotional states and mental health conditions, but its association with loneliness in older adults remains unexplored. In this study, we developed a tablet-based application and collected speech responses of 57 older adults to daily life questions regarding, for example, one's feelings and future travel plans. From audio data of these speech responses, we automatically extracted speech features characterizing acoustic, prosodic, and linguistic aspects, and investigated their associations with self-rated scores of the UCLA Loneliness Scale. Consequently, we found that with increasing loneliness scores, speech responses tended to have less inflections, longer pauses, reduced second formant frequencies, reduced variances of the speech spectrum, more filler words, and fewer positive words. The cross-validation results showed that regression and binary-classification models using speech features could estimate loneliness scores with an R2 of 0.57 and detect individuals with high loneliness scores with 95.6% accuracy, respectively. Our study provides the first empirical results suggesting the possibility of using speech data that can be collected in everyday life for the automatic assessments of loneliness in older adults, which could help develop monitoring technologies for early detection and intervention for mitigating loneliness.

show abstract

Section: Discussionsupporting

confidence: 90%

Section: Methodsmentioning

confidence: 99%

Section: Speech Data Analysismentioning

confidence: 99%

See 1 more Smart Citation

Automatic Assessment of Loneliness in Older Adults Using Speech Analysis on Responses to Daily Life Questions

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Computing Fixed-Length Feature Vectors for Each Video Features extracted from the facial aect, visual, and vocal modalities in each video are frame-by-frame and depend on the length of videos. In order to prepare these features for use in binary deception classication models, we represented each feature as a xed-length vector of statistical attributes that captured the feature's temporal behavior and distribution during the variable-length videos, similar to [43,51]. For each of the facial aect, visual, and vocal features, we used the TsFresh toolkit [11] to compute 130 timeseries attributes, including the following statistical measures: mean, median, standard deviation, skew, kurtosis, maximum, minimum, sum of values, linear trends, autocorrelation with dierent lags, and the changes among values within dierent quantile ranges.…”

Section: 25mentioning

confidence: 99%

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

Mathur

Matarić

2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Automated deception detection systems can enhance health, justice, and security in society by helping humans detect deceivers in highstakes situations across medical and legal domains, among others. Existing machine learning approaches for deception detection have not leveraged dimensional representations of facial aect: valence and arousal. This paper presents a novel analysis of the discriminative power of facial aect for automated deception detection, along with interpretable features from visual, vocal, and verbal modalities. We used a video dataset of people communicating truthfully or deceptively in real-world, high-stakes courtroom situations. We leveraged recent advances in automated emotion recognition in-the-wild by implementing a state-of-the-art deep neural network trained on the A-Wild database to extract continuous representations of facial valence and facial arousal from speakers. We experimented with unimodal Support Vector Machines (SVM) and SVM-based multimodal fusion methods to identify eective features, modalities, and modeling approaches for detecting deception. Unimodal models trained on facial aect achieved an AUC of 80%, and facial aect contributed towards the highest-performing multimodal approach (adaptive boosting) that achieved an AUC of 91% when tested on speakers who were not part of training sets. This approach achieved a higher AUC than existing automated machine learning approaches that used interpretable visual, vocal, and verbal features to detect deception in this dataset, but did not use facial aect. Across all videos, deceptive and truthful speakers exhibited signicant dierences in facial valence and facial arousal, contributing computational support to existing psychological theories on relationships between aect and deception. The demonstrated importance of facial aect in our models informs and motivates the future development of automated, aect-aware machine learning approaches for modeling and detecting deception and other social behaviors in-the-wild.

show abstract

“…Subsequently, there have been efforts (Narayanan and Georgiou, 2013) to automate this behavior annotation (or coding) process using machine learning so that rapid and inexpensive feedback can be provided to the stakeholders. Previous work has shown that automated coding systems are effective at quantifying behaviors from speech and spoken language such as Negativity (Georgiou et al, 2011;Black et al, 2013;Chakravarthula et al, 2015a;Tseng et al, 2017), Depression (Gupta et al, 2014;Morales et al, 2018) and Empathy (Xiao et al, 2012;Gibson et al, 2016;Pérez-Rosas et al, 2017). However, there are some critical aspects of this behavior assessment process which humans can handle naturally and easily but machines still cannot, one of which is the notion of how much to observe in order to reliably assess behavior.…”

Section: Introductionmentioning

confidence: 99%

An analysis of observation length requirements for machine understanding of human behaviors from spoken language

Chakravarthula¹,

Baucom²,

Narayanan³

et al. 2019

Preprint

View full text Add to dashboard Cite

Machine learning-based human behavior modeling, often at the level of characterizing an entire clinical encounter such as a therapy session, has been shown to be useful across a range of domains in psychological research and practice from relationship and family studies to cancer care. Existing approaches typically first quantify the target behavior construct based on cues in an observation window, such as a fixed number of words, and then aggregate it over all the windows in that session. During this process, a sufficiently long window is employed so that adequate information is gathered to accurately estimate the construct. The link between behavior modeling and the observation length, however, has not been well studied, especially for spoken language. In this paper, we analyze the effect of observation window length on the quality of behavior quantification and present a framework for determining appropriate windows for a wide range of behaviors. Our analysis method employs two levels of evaluations: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis on a dataset of real-life married couple interactions that are annotated for a large and diverse set of behavior codes and test the robustness of our findings to different machine learning models. We find that negative constructs such as blame can be accurately identified from short expressions while those pertaining to positive affect such as satisfaction tend to require slightly longer observation windows. Behaviors that describe more complex personality traits such as negotiation and avoidance are found to require very long observations and are difficult to quantify from language alone. Our findings are generally in agreement with similar work using acoustic vocal cues as well as existing literature in psychology on thin slices and human emotion perception.1 we use the term 'behavior' to refer to not just physical actions such as facial expressions, body gestures and speech but the underlying state of mind that is expressed through these actions.

show abstract

A Linguistically-Informed Fusion Approach for Multimodal Depression Detection

Cited by 24 publications

References 21 publications

Automatic Assessment of Loneliness in Older Adults Using Speech Analysis on Responses to Daily Life Questions

Automatic Assessment of Loneliness in Older Adults Using Speech Analysis on Responses to Daily Life Questions

Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

An analysis of observation length requirements for machine understanding of human behaviors from spoken language

Contact Info

Product

Resources

About