Hybrid Acoustic-Lexical Deep Learning Approach for Deception Detection

Abstract

Automatic deception detection is an important problem with far-reaching implications for many disciplines. We present a series of experiments aimed at automatically detecting deception from speech. We use the Columbia X-Cultural Deception (CXD) Corpus, a large-scale corpus of within-subject deceptive and non-deceptive speech, for training and evaluating our models. We compare the use of spectral, acoustic-prosodic, and lexical feature sets, using different machine learning models. Finally, we design a single hybrid deep model with both acoustic and lexical features trained jointly that achieves state-of-the-art results on the CXD corpus.

A Linguistically-Informed Fusion Approach for Multimodal Depression Detection

“…In particular, deep learning approaches to fusion appear to be particularly promising. For example, Mendels et al (2017) presented a single hybrid deep model with both acoustic and lexical features trained jointly and found that this approach to fusion achieved stateof-the-art results for deception detection. However, deep learning is not currently a good approach for depression detection, since labeled corpora are not very large and interpretable models are important.…”

Privacy Implications of Voice and Speech Analysis – Information Disclosure by Inference

“…Research has shown that the veracity of verbal statements can be assessed automatically [60,107]. Among other speech cues, acoustic-prosodic features (e.g., formant frequencies, speech intensity) and lexical features (e.g., verb tense, use of negative emotion words) were found to be predictive of deceptive utterances [67]. Increased changes in speech parameters were observed when speakers are highly motivated to deceive [98].…”

Detecting Concealed Information in Text and Speech

“…Given the results from baseline models and previous literature (Mendels et al, 2017;Levitan et al, 2016), we train Bidirectional Long Short-Term models (BiLSTM, Schuster and Paliwal, 1997;Zhang et al, 2015) with sequences of word embeddings, Multi-Layer Perceptrons (MLP) with acoustic feature sets, and the combinations thereof. The GloVe embeddings were used to initialize the weights but back propagation was also allowed to update embedding values during training.…”

“…and concatenate embeddings learned from acoustic features passed through an MLP and those passed through a BiL-STM for the last softmax layer. The combined model structure follows Mendels et al (2017) except that we used 4 hidden layers for MLP, and concatenated additional individual features before the last softmax layer. Our model consists of four fully connected layers, each with 680 hidden units followed by ReLU (Krizhevsky et al, 2012) activations.…”