Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis

Singh, Nikhil; Mentch, Jeff; Ng, Jerry; Beveridge, Matthew; Drori, Iddo

doi:10.1109/iccv48922.2021.00035

Cited by 31 publications

(17 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However it has higher perceptual errors (RTE and MOSE). Image2Reverb's [55] high errors reveal the difficulty of our task and data, and its inapplicability to AVSpeech highlights our model's self-supervised training advantage. Despite having the estimated RT60 as input (and thus having low RT60 error), Blind Reverberator's STFT and MOS errors are much higher than AViTAR's, showing that images are a promising way to characterize room acoustics beyond the traditional RT60.…”

Section: Results On Soundspaces-speechmentioning

confidence: 99%

“…We demonstrate our approach on challenging real-world sounds and environments, as well as controlled experiments with realistic acoustic simulations in scanned scenes. Our quantitative results and subjective evaluations via human studies show that our model generates audio that matches the target environment with high perceptual quality, outperforming a state-of-the-art model that has heavier supervision requirements [55] as well as traditional acoustic matching models.…”

Section: Source Audiomentioning

confidence: 92%

“…It is therefore inapplicable for the Unseen case (no reference audio) and AVSpeech (no training IRs). [55]. This is a recent approach that trains an IR predictor from images, then convolves the predicted IRs with A S to obtain the target audio.…”

Section: Methodsmentioning

confidence: 99%

“…RIRs are traditionally measured with special equipment in the room itself [28,56] or simulated with sound propagation models [5,12,45]. Recent work explores estimating an RIR from an input image [33,55], which requires access to paired image and impulse response training data. While video recordings provide a natural source for learning the correspondence between space (captured by the visual stream) and acoustics (captured by the audio stream), they have not been explored in the literature.…”

Section: Related Workmentioning

confidence: 99%

“…Importantly, our task formulation does not assume access to the impulse response, nor does it require the input audio to be anechoic. In comparison, the Im-age2Reverb [55] task requires access to both the impulse response and clean input audio, and does not account for the co-location of the camera and microphone.…”

Section: The Visual Acoustic Matching Taskmentioning

confidence: 99%

See 4 more Smart Citations

Visual Acoustic Matching

Chen¹,

Gao²,

Calamia³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment. Given an image of the target environment and a waveform for the source audio, the goal is to re-synthesize the audio to match the target room acoustics as suggested by its visible geometry and materials. To address this novel task, we propose a cross-modal transformer model that uses audio-visual attention to inject visual properties into the audio and generate realistic audio output. In addition, we devise a self-supervised training objective that can learn acoustic matching from in-the-wild Web videos, despite their lack of acoustically mismatched audio. We demonstrate that our approach successfully translates human speech to a variety of real-world environments depicted in images, outperforming both traditional acoustic matching and more heavily supervised baselines.

show abstract

Section: Results On Soundspaces-speechmentioning

confidence: 99%

Section: Source Audiomentioning

confidence: 92%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: The Visual Acoustic Matching Taskmentioning

confidence: 99%

See 3 more Smart Citations

Visual Acoustic Matching

Chen¹,

Gao²,

Calamia³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

UnderwaterImage2IR: Underwater impulse response generation via dual‐path pre‐trained networks and conditional generative adversarial networks

Zhang,

Liu

2024

Computer Animation & Virtual

View full text Add to dashboard Cite

In the field of acoustic simulation, methods that are widely applied and have been proven to be highly effective rely on accurately capturing the impulse response (IR) and its convolution relationship. This article introduces a novel approach, named as UnderwaterImage2IR, that generates acoustic IRs from underwater images using dual‐path pre‐trained networks. This technique aims to achieve cross‐modal conversion from underwater visual images to acoustic information with high accuracy at a low cost. Our method utilizes deep learning technology by integrating dual‐path pre‐trained networks and conditional generative adversarial networks conditional generative adversarial networks (CGANs) to generate acoustic IRs that match the observed scenes. One branch of the network focuses on the extraction of spatial features from images, while the other is dedicated to recognizing underwater characteristics. These features are fed into the CGAN network, which is trained to generate acoustic IRs corresponding to the observed scenes, thereby achieving high‐accuracy acoustic simulation in an efficient manner. Experimental results, compared with the ground truth and evaluated by human experts, demonstrate the significant advantages of our method in generating underwater acoustic IRs, further proving its potential application in underwater acoustic simulation.

show abstract