Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

Abstract

Abstract-In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia-the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener. We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by the loudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented.

Spatial Soundscapes and Virtual Worlds: Challenges and Opportunities

“…They also noted that localization can be improved if there is some regularity in the distribution of the sound sources. Kose et al (2018) ran a similar experiment, simultaneously presenting acoustic and visual stimuli in a VR environment. They used a single speaker moving along a rectangular path.…”

“…Kose et al ( 2018 ) ran a similar experiment, simultaneously presenting acoustic and visual stimuli in a VR environment. They used a single speaker moving along a rectangular path.…”

A Method of Verifying the Robot's Trajectory for Goals With a Shared Workspace

“…VR simulation could also be used in manufacturing, robotics and control systems (Chen et al, 2018;Covaciu et al, 2018;Kot, Novak & Bajak, 2018;Oyekan et al, 2019) by simulating different algorithms and control methods. Another application is audioto use VR as a sound visualization tool for artists or the blind (Kose et al, 2018). Moreover, smart racks and simulations are done with force feedback (Burdea, 1999) for a better haptic feeling of tested in VR joysticks, devices or manipulators (Kuts, Otto, Tahemaa & Bondarenko, 2019).…”

Digital Twin Based Synchronised Control and Simulation Of the Industrial Robotic Cell Using Virtual Reality

“…Manufacturing, robotics, and control Architecture [5] of varying scale rooms and buildings, for finding the most efficient way of design. Audio [6] to use VR as a sound visualisation tool for the artist or blind people. Moreover, smart racks and simulations are done with force feedback [7,8] for a better haptic feeling of tested in VR joysticks, devices or manipulators.…”