TIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection

Abstract

With the rapid development of deep learning techniques, the generation and counterfeiting of multimedia material has become increasingly simple. Current technology enables the creation of videos where both the visual and audio contents are falsified. While the multimedia forensics community has begun to address this threat by developing fake media detectors. However, the vast majority existing forensic techniques only analyze one modality at a time. This is an important limitation when authenticating manipulated videos, because sophisticated forgeries may be difficult to detect without exploiting cross-modal inconsistencies (e.g., across the audio and visual tracks). One important reason for the lack of multimodal detectors is a similar lack of research datasets containing multimodal forgeries. Existing datasets typically contain only one falsified modality, such as deepfaked videos with authentic audio tracks, or synthetic audio with no associated video. Currently, datasets are needed that can be used to develop, train, and test these forensic algorithms. In this paper, we propose a new audio-visual deepfake dataset containing multimodal video forgeries. We present a general pipeline for synthesizing deepfake speech content from a given video, facilitating the creation of counterfeit multimodal material. The proposed method uses Text-to-Speech (TTS) and Dynamic Time Warping (DTW) techniques to achieve realistic speech tracks. We use this pipeline to generate and release TIMIT-TTS, a synthetic speech dataset containing the most cutting-edge methods in the TTS field. This can be used as a standalone audio dataset, or combined with DeepfakeTIMIT and VidTIMIT video datasets to perform multimodal research. Finally, we present numerous experiments to benchmark the proposed dataset in both monomodal (i.e., audio) and multimodal (i.e., audio and video) conditions. This highlights the need for multimodal forensic detectors and more multimodal deepfake data.

A Robust Approach to Multimodal Deepfake Detection

“…TIMIT-TTS [ 6 ]. This is a speech dataset including only fake audio samples, generated starting from the VidTIMIT corpus.…”

“…In the recent years, few multimodal datasets have been proposed, containing both counterfeited video and audio tracks. These are DFDC [ 42 ], FakeAVCeleb [ 43 ], and DeepfakeTIMIT [ 44 ] with TIMIT-TTS [ 6 ]. In the following sections, we provide further details on these datasets and test our proposed multimodal detector on them.…”

“…In our experiments, we merge LQ and HQ subsets, considering them as a unique corpus. TIMIT-TTS [6]. This is a speech dataset including only fake audio samples, generated starting from the VidTIMIT corpus.…”

“…Visual-only detectors, for example, can be deceived by audio deepfakes, while audio-only detectors are vulnerable to deepfakes that manipulate visual content [ 6 ]. Furthermore, some information is lost during these analyses, such as the consistency between modalities, which is sometimes crucial for detecting synthetic content.…”

Towards Frequency Band Explainability in Synthetic Speech Detection

“…On the other hand, the methods proposed in [6] and [7] perform synthetic speech detection by analyzing the emotional and prosodic content of speech. Also, numerous datasets have been presented in this field to increase the interest of the scientific community on the topic and push the research toward the development of new detection methods [8], [9].…”