Do Deepfakes Adequately Display Emotions? A Study on Deepfake Facial Emotion Expression

Abstract

Recent technological advancements in Artificial Intelligence make it easy to create deepfakes and hyper-realistic videos, in which images and video clips are processed to create fake videos that appear authentic. Many of them are based on swapping faces without the consent of the person whose appearance and voice are used. As emotions are inherent in human communication, studying how deepfakes transfer emotional expressions from original to fakes is relevant. In this work, we conduct an in-depth study on facial emotional expression in deepfakes using a well-known face swap-based deepfake database. Firstly, we extracted the photograms from their videos. Then, we analyzed the emotional expression in the original and faked versions of video recordings for all performers in the database. Results show that emotional expressions are not adequately transferred between original recordings and the deepfakes created from them. High variability in emotions and performers detected between original and fake recordings indicates that performer emotion expressiveness should be considered for better deepfake generation or detection.

Pengembangan Aplikasi Emoticon Recognition dan Facial Recognition menggunakan Algoritma Local Binary Pattern Histogram (LBPH) dan Convolutional Neural Network (CNN)

“…Salah satu cara yang dapat digunakan untuk mendeteksi sebuah deepfake adalah dengan menggunakan pendeteksi emosi atau emotion recognition. Dikarenakan sebuah deepfake yang diproduksi dengan menggunakan algoritma face-swap belum dapat menciptakan kembali ekspresi yang sesuai dengan emosi [8]. Kemudian lagi sebuah deepfake juga belum bisa menghasilkan ekspresi emosi yang konsisten dengan emosi yang dideteksi dari dalam suaranya [9].…”

“…Pada tahap ini penulis melakukan studi literatur dan mendapatkan bahwa deepfake sebagai salah satu kelemahan dari teknologi facial recognition dan menemukan cara untuk mengatasi masalah tersebut. Berdasarkan [8], [9] emosi dapat digunakan sebagai sebuah cara untuk mendeteksi deepfake mulai dari kualitas yang rendah sampai dengan kualitas yang tinggi.…”

Analyzing the Legal and Ethical Considerations of Deepfake Technology

“…Langer and Wyczik (2020) provide a legal analysis of deepfakes, emphasizing the inadequacy of current laws in addressing the multifaceted challenges posed by deepfakes (Langer & Wyczik, 2020). López-Gil, Gil, and García (2022) contribute to the understanding of emotional expression in deepfakes, an aspect critical to their believability and potential for harm (López-Gil et al, 2022). Meškys et al (2020) call for a nuanced approach to regulating deepfakes, balancing legal and ethical considerations to foster innovation while protecting against abuse (Meškys et al, 2020).…”