Mixed Source Sound Field Translation for Virtual Binaural Application With Perceptual Validation

Birnie, Lachlan; Abhayapala, Thushara D.; Tourbabin, Vladimir; Samarasinghe, Prasanga N.

doi:10.1109/taslp.2021.3061939

Cited by 24 publications

(23 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The essential hardware of Metaverse is an HMD that blocks the view to enable immersive participation. For a more effective visual experience, Birnie et al [64] proposed a fovea rendering method that maintains the central part in high resolution similar to human vision. Critical factors for physical devices and sensors are resolution, the field of view size, and latency.…”

Section: Ryskeldiev Et Al [48]mentioning

confidence: 99%

“…However, in Metaverse, users can change their listening position as they run, spin, or various body changes. Birnie et al [64] proposed a method for binaural playback of microphone recordings in a virtual application in which one's body freely moves beyond the recording location. They integrate near, and far sources in an extended virtual environment and better reproduce the intensity and binaural room impulse response spectrum of the near environment.…”

Section: Sound and Speech Recognitionmentioning

confidence: 99%

See 1 more Smart Citation

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

2022

View full text Add to dashboard Cite

Unlike previous studies on the Metaverse based on Second Life, the current Metaverse is based on the social value of Generation Z that online and offline selves are not different. With the technological development of deep learning-based high-precision recognition models and natural generation models, Metaverse is being strengthened with various factors, from mobile-based always-on access to connectivity with reality using virtual currency. The integration of enhanced social activities and neural-net methods requires a new definition of Metaverse suitable for the present, different from the previous Metaverse. This paper divides the concepts and essential techniques necessary for realizing the Metaverse into three components (i.e., hardware, software, and contents) and three approaches (i.e., user interaction, implementation, and application) rather than marketing or hardware approach to conduct a comprehensive analysis. Furthermore, we describe essential methods based on three components and techniques to Metaverse's representative Ready Player One, Roblox, and Facebook research in the domain of films, games, and studies. Finally, we summarize the limitations and directions for implementing the immersive Metaverse as social influences, constraints, and open challenges.

show abstract

Section: Ryskeldiev Et Al [48]mentioning

confidence: 99%

Section: Sound and Speech Recognitionmentioning

confidence: 99%

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

2022

View full text Add to dashboard Cite

show abstract

“…Nevertheless, recent studies have shown promising progress by employing dual-band approaches and heuristics to match early reflections in the time domain [58,59]. On a related note, another active research topic is the extrapolation of RIRs in the Ambisonics domain for 6DoF applications (e.g., [60][61][62][63]), which is further discussed in Section 4.…”

Section: The Binaural Casementioning

confidence: 99%

“…Several methods have been recently proposed to efficiently extrapolate spatial audio signals from one listener position to another, either via simple parametric methods [76]. [87] or more complex Ambisonics-based approaches that often rely on parametrising the sound field in 'direct' and 'ambient' components [60,61], or according to the source distance [62,63]. Significant advancements have also been made in terms of recording complex auditory scenes and to make them navigable in 6DoF-in this case, specialised hardware and software has been released and is already available commercially [88].…”

Section: Future Directionsmentioning

confidence: 99%

Reverberation and its Binaural Reproduction: The Trade-off between Computational Efficiency and Perceived Quality

Engel¹,

Picinali²

2022

Advances in Fundamental and Applied Research on Spatial Audio

View full text Add to dashboard Cite

Accurately rendering reverberation is critical to produce realistic binaural audio, particularly in augmented reality applications where virtual objects must blend in seamlessly with real ones. However, rigorously simulating sound waves interacting with the auralised space can be computationally costly, sometimes to the point of being unfeasible in real time applications on resource-limited mobile platforms. Luckily, knowledge of auditory perception can be leveraged to make computational savings without compromising quality. This chapter reviews different approaches and methods for rendering binaural reverberation efficiently, focusing specifically on Ambisonics-based techniques aimed at reducing the spatial resolution of late reverberation components. Potential future research directions in this area are also discussed.

show abstract

“…Hence, the target of these techniques is to convey an effective spatial audio perception to the user rather than a perfect reconstruction of the sound field. On the other hand, non-parametric methods [ 16 , 17 , 18 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] aim to numerically estimate the acoustic field. In this category, the sound field is typically modelled as a linear combination of solutions of the wave equation [ 27 ], e.g., plane wave [ 25 , 26 ] or spherical wave [ 22 , 23 , 24 , 28 , 29 ] expansions.…”

Section: Introductionmentioning

confidence: 99%

Deep Prior Approach for Room Impulse Response Reconstruction

Pezzoli

Perini

Bernardini

et al. 2022

Sensors

View full text Add to dashboard Cite

In this paper, we propose a data-driven approach for the reconstruction of unknown room impulse responses (RIRs) based on the deep prior paradigm. We formulate RIR reconstruction as an inverse problem. More specifically, a convolutional neural network (CNN) is employed prior, in order to obtain a regularized solution to the RIR reconstruction problem for uniform linear arrays. This approach allows us to avoid assumptions on sound wave propagation, acoustic environment, or measuring setting made in state-of-the-art RIR reconstruction algorithms. Moreover, differently from classical deep learning solutions in the literature, the deep prior approach employs a per-element training. Therefore, the proposed method does not require training data sets, and it can be applied to RIRs independently from available data or environments. Results on simulated data demonstrate that the proposed technique is able to provide accurate results in a wide range of scenarios, including variable direction of arrival of the source, room T60, and SNR at the sensors. The devised technique is also applied to real measurements, resulting in accurate RIR reconstruction and robustness to noise compared to state-of-the-art solutions.

show abstract

Mixed Source Sound Field Translation for Virtual Binaural Application With Perceptual Validation

Cited by 24 publications

References 44 publications

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

Reverberation and its Binaural Reproduction: The Trade-off between Computational Efficiency and Perceived Quality

Deep Prior Approach for Room Impulse Response Reconstruction

Contact Info

Product

Resources

About