Audio-Visual Inpainting: Reconstructing Missing Visual Information with Sound

Sanguineti, Valentina; Thakur, Sanket; Morerio, Pietro; Bue, Alessio Del; Murino, Vittorio

doi:10.1109/icassp49357.2023.10095447

Cited by 1 publication

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sanguineti et al [29] conducted a study on multi-modal image generation for audiovisual inpainting. Yet, their model is also capable of performing cross-modal generation conditioned on sound only.…”

Section: Related Workmentioning

confidence: 99%

Sound-to-Imagination: An Exploratory Study on Cross-Modal Translation Using Diverse Audiovisual Data

Fanzeres,

Nadeu

2023

Applied Sciences

View full text Add to dashboard Cite

The motivation of our research is to explore the possibilities of automatic sound-to-image (S2I) translation for enabling a human receiver to visually infer occurrences of sound-related events. We expect the computer to ‘imagine’ scenes from captured sounds, generating original images that depict the sound-emitting sources. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or supervision/self-supervision for training. In contrast, our approach involves performing S2I translation using thousands of distinct and unknown scenes, using sound class annotations solely for data preparation, just enough to ensure aural–visual semantic coherence. To model the translator, we employ an audio encoder and a conditional generative adversarial network (GAN) with a deep densely connected generator. Furthermore, we present a solution using informativity classifiers for quantitatively evaluating the generated images. This allows us to analyze the influence of network-bottleneck variation on the translation process, highlighting a potential trade-off between informativity and pixel space convergence. Despite the complexity of the specified S2I translation task, we were able to generalize the model enough to obtain more than 14%, on average, of interpretable and semantically coherent images translated from unknown sounds.

show abstract

Section: Related Workmentioning

confidence: 99%