Generating Visually Aligned Sound From Videos

Abstract

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated outside a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and contentwise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet.

Collaborative Learning to Generate Audio-Video Jointly

“…Another similar work [27] generated music for a given video. [28] generated visually aligned sounds from videos. In sound2sight [29], future video frames and motion dynamics are generated by conditioning on audio and a few past frames.…”

Audio-Visual Model for Generating Eating Sounds Using Food ASMR Videos

“…Chen et al [5] exploited conditional generative adversarial networks to generate cross-modal audio visuals of musical performances. Chen et al [6] designed an audio forwarding regularizer that could control the irrelevant sound component, thereby preventing the model from learning incorrect mapping between the video frames and the sound emitted by the out-ofscreen objects. Akbari et al [3] tried to reconstruct natural sounding speech using a neural network that takes as input the face region of the talker and estimates bottleneck features extracted from the auditory spectrogram by a pre-trained autoencoder.…”

BiSPL: Bidirectional Self-Paced Learning for Recognition From Web Data

“…For example, the well-known MoCo [33], Sim-CLR [34], and BYOL [35] perform basic data augmentation operations on unlabeled images to obtain anchors, positive examples and negative examples, and then minimize the distances between positive pairs while maximizing those between negative pairs. In addition, some studies also design various pretext tasks such as image colorization [36], jigsaw puzzles [37], and image inpainting [38] for unlabeled image data and vehicle tracking [39], relative speed perception [40], background erasure [41], and sound generation [42] for unlabeled audio and video data. Based on models trained on pretext tasks, self-supervised learning can further transfer the learned features to downstream tasks.…”