2.5D Visual Sound

Abstract

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound-the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs. utexas.edu/projects/2.5D_visual_sound/ In

Co-Separating Sounds of Visual Objects

“…Audio-Visual Source Separation Early methods for audio-visual source separation focus on mutual information [10], subspace analysis [42,34], matrix factorization [33,39], and correlated onsets [5,27]. Recent methods leverage deep learning for separating speech [8,31,3,11], musical instruments [52,13,51], and other objects [12]. Similar to the audio-only methods, almost all use a "mixand-separate" training paradigm to perform video-level separation by artificially mixing training videos.…”

“…Generating Sounds from Video Sound generation methods synthesize a sound track from a visual input [32,54,6]. Given both visual input and monaural audio, recent methods generate spatial (binaural or ambisonic) audio [13,30]. Unlike any of the above, our work aims to separate an existing real audio track, not synthesize plausible new sounds.…”

“…Following [21,18,50,52,31,13,8], we start with the commonly adopted "mix-and-separate" idea to selfsupervise source separation. Given two training videos V 1 and V 2 with corresponding audios x 1 (t) and x 2 (t), we use a pre-trained object detector to find objects in both videos.…”

“…2 illustrates our audio-visual separator network that performs audio-visual feature aggregation and source separation. A related design for multi-modal feature fusion is also used in [13,30,31] for audio spatialization and separation. However, unlike those models, our separator network combines the visual features of a localized object region and the audio features of the mixed audio to predict a magnitude spectrogram mask for source separation.…”

Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

“…These may imply that, by the definition of learnability [12], the task is not a fully learnable problem only with unsupervised data in our setting, which is static-image based single-channel audio source localization, but can be fixed with even a small amount of relevant prior knowledge. Although the sound localization task is not effectively addressed with our unsupervised learning approach with static images and mono audios, other methods that use spatial microphones [25], [53], [54], [55] or temporal information, motion [8] and synchronization [18], with multiple frames have been shown to perform well on this task with unsupervised algorithms. In the following, we conclude our work with additional discussion for future investigation.…”