Real-time 3-D Mapping with Estimating Acoustic Materials

Abstract

This paper proposes a real-time system integrating an acoustic material estimation from visual appearance and an on-the-fly mapping in the 3-dimension. The proposed method estimates the acoustic materials of surroundings in indoor scenes and incorporates them to a 3-D occupancy map, as a robot moves around the environment. To estimate the acoustic material from the visual cue, we apply the state-of-the-art semantic segmentation CNN network based on the assumption that the visual appearance and the acoustic materials have a strong association. Furthermore, we introduce an update policy to handle the material estimations during the online mapping process. As a result, our environment map with acoustic material can be used for sound-related robotics applications, such as sound source localization taking into account various acoustic propagation (e.g., reflection).

Perceptual Matching of Room Acoustics for Auditory Augmented Reality in Small Rooms - Literature Review and Theoretical Framework

“…The available information can be acoustic impulse responses, measured with a single (omni-directional) microphone (Pörschmann et al, 2017), a head-and-torso-simulator (Sloma et al, 2019; Garcia-Gomez & Lopez, 2018) or microphone array solutions (Garí et al, 2019; Stade, 2018; Zaunschirm et al, 2020; Müller & Zotter, 2020; McCormack et al, 2020; Engel & Picinali, 2022). Besides, for example, semantic and visual information can be used to estimate acoustic properties (Kim et al, 2017, 2020). BRIR synthesis can be realized either by pure simulation, for example, based on ray-tracing (Savioja & Svensson, 2015; Brinkmann et al, 2019), wave-based simulation approaches or delay networks (Alary et al, 2019; Välimäki et al, 2012) or by manipulation of measured impulse responses, like interpolation (Bruschi et al, 2020; Brinkmann et al, 2020), extrapolation (Neidhardt et al, 2018; Sloma et al, 2019; Coleman et al, 2017; Pörschmann et al, 2017) or shaping of the late reverberation tail (Jot & Lee, 2016; Pörschmann & Zebisch, 2012; Arend et al, 2021).…”