Deepfake Speech Detection Through Emotion Recognition: A Semantic Approach

Abstract

In recent years, audio and video deepfake technology has advanced relentlessly, severely impacting people's reputation and reliability. Several factors have facilitated the growing deepfake threat. On the one hand, the hyper-connected society of social and mass media enables the spread of multimedia content worldwide in real-time, facilitating the dissemination of counterfeit material. On the other hand, neural network-based techniques have made deepfakes easier to produce and difficult to detect, showing that the analysis of low-level features is no longer sufficient for the task. This situation makes it crucial to design systems that allow detecting deepfakes at both video and audio levels. In this paper, we propose a new audio spoofing detection system leveraging emotional features. The rationale behind the proposed method is that audio deepfake techniques cannot correctly synthesize natural emotional behavior. Therefore, we feed our deepfake detector with high-level features obtained from a state-of-the-art Speech Emotion Recognition (SER) system. As the used descriptors capture semantic audio information, the proposed system proves robust in cross-dataset scenarios outperforming the considered baseline on multiple datasets.

Do Deepfakes Adequately Display Emotions? A Study on Deepfake Facial Emotion Expression

“…Works on how to include emotional cues in deepfake detection are starting to be Columns display the number of original recordings by the performer (OR), the number of performers of deepfakes on the original recordings of the current performer (PF), and number of fake recordings made on recordings by the performer (FRP). en, the number of photograms with emotions in common with performed [15][16][17], which indicates that emotional cues in video recordings can be used effectively for deepfake detection.…”

“…Consequently, the effect of bad neutral photograms in fakes, mainly in initial photograms, may suggest that emotion recognition is prone to work worse than in their corresponding original recordings. Works on how to include emotional cues in deepfake detection are starting to be performed [ 15 – 17 ], which indicates that emotional cues in video recordings can be used effectively for deepfake detection.…”

“…Literature on deepfakes and emotions is scarce. To the authors' knowledge, the only works that specifically tackle audio-visual deepfake detection using affective cues are [ 15 , 16 ], and [ 17 ]. Moreover, we could not find specific studies on how well deepfakes express emotions.…”

“…To the authors' knowledge, [ 15 , 16 ], and [ 17 ] are the only studies that exclusively address audio-visual deepfake detection using emotive cues. To detect falsifications or manipulations in the input video, the approach provided in [ 15 ] concurrently utilizes the audio (speech) and video (facial) modalities, as well as perceived emotion components collected from both modalities.…”

“…They used both modalities to detect similarity (or dissimilarity) between modality signals, and they discovered that perceived emotional information aids in detecting deepfake content. [ 16 , 17 ] offered a technique for detecting deepfakes based on semantic consistency in emotion, which was based on previous emotion identification work that extracted emotions over time from a subject's speech and faces separately. Synthesized voices or faces are then detected by analyzing these emotional signals.…”