Multimodal Interaction of Contextual and Non-Contextual Sound and Haptics in Virtual Simulations

Abstract

Touch plays a fundamental role in our daily interactions, allowing us to interact with and perceive objects and their spatial properties. Despite its importance in the real-world, touch is often ignored in virtual environments. However, accurately simulating the sense of touch is difficult, requiring the use of high-fidelity haptic devices that are cost-prohibitive. Lower fidelity consumer-level haptic devices are becoming more widespread, yet are generally limited in perceived fidelity and the range of motion (degrees of freedom) required to realistically simulate many tasks. Studies into sound and vision suggest that the presence or absence of sound can influence task performance. Here, we explore whether the presence or absence of contextually relevant sound cues influences the performance of a simple haptic drilling task. Although the results of this study do not show any statistically significant difference in task performance with general (task-irrelevant) sound, we discuss how this is a necessary step in understanding the role of sound on haptic perception.

The Influence of Auditory Cues on Bodily and Movement Perception

“…Such differences in induced perceptions of movement between the real and virtual surfaces may reflect the congruence of auditory, tactile, and proprioceptive information in the real surface condition vs. relative incongruence in the virtual surface condition (i.e., no tactile input paired with auditory cues). Similarly, it was found that when performing virtual drilling (holding a haptic device), there was no effect of naturalistic, contextually relevant sound (drilling sound) or non-naturalistic sound (white noise or classical music) on the perception of drilling depth (Melaisi et al, 2018). Lastly, the effect of sound on perceived exertion during exercise has also been extensively explored, but less so when auditory cues are purposefully inaccurate.…”

Audiohaptic Feedback Enhances Motor Performance in a Low-Fidelity Simulated Drilling Task

“…The drilling sounds used in the simulation were actual recordings from a Stanley Black and Decker (New Britain, CT, USA) consumer drill, drilling in open air and drilling through wood. The recordings were made in an audiometric room to limit external noise and reverberation of sounds within the environment [11]. Two separate auditory stimuli were used in every trial; an initial "air" recording, representing a drill operating in open air, followed by a "wood" audio clip, recorded from the sounds made from drilling through a block of wood.…”

“…Higher-fidelity haptic devices can typically provide up to 40 N of force, with five to seven degrees of freedom [3]. As described in [11], the haptic sensations provided in this simulation were modeled after the resulting forces and vibrations measured from a real drill drilling through a piece of wood. Haptic forces were simulated using a spring-mass system coded into the Unity game engine (Unity Technologies, San Francisco, CA, USA).…”

“…We are constantly engaging with various stimuli, and when two or more sensory inputs are congruent and presented simultaneously, we are able to process and respond to our environment more quickly than if the stimuli were presented separately [15,23]. Visual and auditory interactions are among the most studied, however, there is an increase in research investigating the role of technology in human-haptic interactions [3,10,11]. Sound can induce an intimate response to our physiology, as discussed in a recent study that found an increase in cortisol levels during video game play with music, as compared to the same game without music [24].…”

“…Sound can induce an intimate response to our physiology, as discussed in a recent study that found an increase in cortisol levels during video game play with music, as compared to the same game without music [24]. Manipulating auditory stimuli in video games can also influence a user's play experience, which ultimately affects their feelings of presence [11,25]. Presence refers to the feeling of being situated within a virtual environment regardless of the user's physical location, and is thought to be conducive to effective performance in the virtual world [26].…”