Generative Modeling of Audible Shapes for Object Perception

Abstract

Humans infer rich knowledge of objects from both auditory and visual cues. Building a machine of such competency, however, is very challenging, due to the great difficulty in capturing large-scale, clean data of objects with both their appearance and the sound they make. In this paper, we present a novel, open-source pipeline that generates audiovisual data, purely from 3D object shapes and their physical properties. Through comparison with audio recordings and human behavioral studies, we validate the accuracy of the sounds it generates. Using this generative model, we are able to construct a synthetic audio-visual dataset, namely Sound-20K, for object perception tasks. We demonstrate that auditory and visual information play complementary roles in object perception, and further, that the representation learned on synthetic audio-visual data can transfer to real-world scenarios.

2.5D Visual Sound

“…Material properties are revealed by the sounds objects make when hit with a drumstick, and can be used to synthesize new sounds from silent videos [33]. Recurrent networks [54] or conditional generative adversarial networks [7] can generate audio for input video frames, while powerful simulators can synthesize audio-visual data for 3D shapes [52]. Rather than generate audio from scratch, our task entails converting an input one-channel audio to twochannel binaural audio guided by the visual frames.…”

Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

“…To calculate this function, tradition way is using fast multi-pole method. [26] But the computational cost of this method is very high. So we use finite difference time domain method mentioned in [23].…”

Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction

“…Previous work has used sound to better understand objects in scenes. For instance, impact sounds from interacting with objects in a scene to perform segmentation ] and to emulate the sensory interactions of human information processing [Zhang et al 2017a]. Audio has also been used to compute material [Ren et al 2013], object [Zhang et al 2017a], scene [Schissler et al 2018], and acoustical [Tang et al 2020] properties.…”