FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

An Overview of Visual Sound Synthesis Generation Tasks Based on Deep Learning Networks

“…Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN): CNN and RNN, as fundamental deep learning models, are widely used in Foley production. For instance, Andrew Owens in [17] employed a trained CNN to predict the sounds that occurred during video recording. Understanding these basic neural network models provides a profound insight into the related algorithmic models.…”

“…This evaluation revealed that the author's binary-based model approach had the most significant advantages. Additionally, [17] and [13] both used the Inception Score (IS) for assessment. The core idea of IS involves utilizing a pretrained image classification model, such as the Inception network, to classify the generated images.…”

“…Additionally, they proposed research on combining Internet of Things (IoT) with cloud computing as an emerging concept. Following a similar research approach in 2021, the same authors combined TRN and GAN to redesign a model and published it in [17]. Their aim was also to address the consistency between video and audio content in Foley content generation.…”

“…In [17], the CMCGAN framework was proposed, which unifies four cross-modal generation methods for videos and sounds (bidirectional generation of sound and images). The four models share the same matrix weights, and input content from both sound sources and image sources is processed through cross-computation.…”

“…In subsequent research, in [17], a model called POCAN was introduced by combining CNN and LSTM. In this model, a frame from the video is processed through CNN to extract basic image features, which are then input into an LSTM for category classification.…”