Pros and cons of comparing and combining hand-crafted and neural network based DeepFake detection based on eye blinking behavior

Abstract

Temporal feature spaces are a promising approach for Deep-Fake detection, since DeepFake synthesis is most often done on a frame-by-frame basis. With the existing and upcoming regulations on European level, the EU General Data Protection Regulation (EU GDPR) and Artificial Intelligence Act (AIA) in particular, data minimization and decision transparency are of concern also for such media forensic methods. In order to bring these aspects together, this paper utilizes two different algorithms both analyzing the eye blinking in the videos. The first one is implemented using deep learning to predict blinking behavior. It shows challenges of hyper-parameter tuning for the training of such a model. The second detector uses an existing hand-crafted approach to identify a suitable number of frames (i.e., video duration) required to reliably detect DeepFakes. Considering GDPR concerns, an optimal trade-off between detection performance and data minimization is found in the range of 35 to 40 seconds of video, giving a detection accuracy of 96.88% for the DeepFakes tested.