Lie Recognition with Multi-Modal Spatial–Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory

Abstract

Recognition of lying is a more complex cognitive process than truth-telling because of the presence of involuntary cognitive cues that are useful to lie recognition. Researchers have proposed different approaches in the literature to solve the problem of lie recognition from either handcrafted and/or automatic lie features during court trials and police interrogations. Unfortunately, due to the cognitive complexity and the lack of involuntary cues related to lying features, the performances of these approaches suffer and their generalization ability is limited. To improve performance, this study proposed state transition patterns based on hands, body motions, and eye blinking features from real-life court trial videos. Each video frame is represented according to a computed threshold value among neighboring pixels to extract spatial–temporal state transition patterns (STSTP) of the hand and face poses as involuntary cues using fully connected convolution neural network layers optimized with the weights of ResNet-152 learning. In addition, this study computed an eye aspect ratio model to obtain eye blinking features. These features were fused together as a single multi-modal STSTP feature model. The model was built using the enhanced calculated weight of bidirectional long short-term memory. The proposed approach was evaluated by comparing its performance with current state-of-the-art methods. It was found that the proposed approach improves the performance of detecting lies.

Translated Pattern-Based Eye-Writing Recognition Using Dilated Causal Convolution Network

“…The Pf identifies a character that the network recognized as belonging to a wrong eye-writing character class that does not. The Nt refers to the eyewriting characters that the network correctly recognized as wrong ones [56]. Thus, the confusion matrix can be decomposed to obtain the average accuracy, precision, recall, and f1-score using eq.…”

Application of Intelligent Lie Recognition Technology in Laws and Regulations Based on Occupational Mental Health Protection

“…Abdullahi proposed a spatiotemporal local binary integral map mode to improve the accuracy of micro-expression recognition by extracting the integral projection of facial difference images in the horizontal and vertical directions. 7 Abdullahi proposed a sparse promoted dynamic mode decomposition to eliminate the redundancy problem of micro-expression motion information caused by high-speed cameras, so that it could extract representative dynamic changes and improve the micro-expression recognition rate. This mode explored the changes in micro-expressions from the space-time texture feature from the perspective of the video sequence.…”