Natural Listening over Headphones in Augmented Reality Using Adaptive Filtering Techniques

Abstract

Augmented reality (AR), which composes of virtual and real world environments, is becoming one of the major topics of research interest due to the advent of wearable devices. Today, AR is commonly used as assistive display to enhance the perception of reality in education, gaming, navigation, sports, entertainment, simulators, etc. However, most of the past works have mainly concentrated on the visual aspects of AR. Auditory events are one of the essential components in human perceptions in daily life but the augmented reality solutions have been lacking in this regard till now compared to visual aspects. Therefore, there is a need of natural listening in AR systems to give a holistic experience to the user. A new headphones configuration is presented in this work with two pairs of binaural microphones attached to headphones (one internal and one external microphone on each side). This paper focuses on enabling natural listening using open headphones employing adaptive filtering techniques to equalize the headset such that virtual sources are perceived as close as possible to sounds emanating from the physical sources. This would also require a superposition of virtual sources with the physical sound sources, as well as ambience. Modified versions of the filtered-x normalized least mean square algorithm (FxNLMS) are proposed in the paper to converge faster to the optimum solution as compared to the conventional FxNLMS. Measurements are carried out with open structure type headphones to evaluate their performance. Subjective test was conducted using individualized binaural room impulse responses (BRIRs) to evaluate the perceptual similarity between real and virtual sounds.Index Terms-Adaptive filtering, augmented reality (AR), head related transfer function (HRTF), natural listening, spatial audio. Manuscript

Spatial Audio for Soundscape Design: Recording and Reproduction

“…The characteristics of the acoustic environment are summarized in Table 3. [27,97] The acoustic environment characteristics are organized firstly by its type, namely: (1) "Virtual(R)", a simulated environment reproduced from recordings at a real location; (2) "Virtual(S)", a simulated environment produced by auralization [81]; and (3) "Real", the actual physical environment.…”

“…The sound can be rendered using either a pair of headphones or an array of loudspeakers arranged in a specific configuration. The aim of the system is to reproduce the sound field in such a manner so as to give a sense of perception of spaciousness and directivity of sound objects located in 3D space [27]. The sounds generated in such a scenario are often referred to as virtual sound in contrast to real sound we commonly hear in everyday life [59].…”

“…In 2015, MPEG-H standard was announced, with the aim to ensure all types of sound formats are supported in any types of playback systems. The most recent attempts of spatial audio rendering were focused on VR/AR applications [27,62].…”

“…Tikander et al [96] also developed a mixer for equalizing and mixing virtual objects with real environment. Ranjan et al [27] proposed an augmented reality headset in which they use open-back headphones with pairs of external and internal microphones to achieve sound playback that is very similar to natural listening scenario.…”

“…Recently, virtual reality (VR) [23][24][25] and augmented reality (AR) [26][27][28] technologies have been increasingly adopted in soundscape studies due to their potential in creating a perceptually accurate audio-visual scene. To create virtual acoustic environments with high ecological validity, it is essential to have a holistic understanding of recording and reproducing techniques with respect to soundscape evaluation methodologies.…”