Towards Multisensory Perception: Modeling and Rendering Sounds of Tool-Surface Interactions

Lu, Shihan; Chen, Yang; Culbertson, Heather

doi:10.1109/toh.2020.2966192

Cited by 10 publications

(15 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[25] Accordingly, there is an obvious need and opportunity to leverage vibration cues in tactile pseudo-haptics for rendering manual interactions with virtual objects. Moreover, multisensory pseudo-haptics may further benefit from the addition of auditory feedback, which conveys rich information about object-based interactions [26] and systematically shapes tactile vibration perception. [27,28] Finally, our experiments explored only how user performance with the combination of referred haptic feedback at the wrist and visual pseudo-haptics compares to each modality alone.…”

Section: Discussionmentioning

confidence: 99%

Multisensory Pseudo‐Haptics for Rendering Manual Interactions with Virtual Objects

Pezent

Macklin

Yau

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Over the past decade, advances in optics, displays, graphics, tracking, environment mapping, and audio have revolutionized technologies for extended reality (XR). The scope of XR has exploded in recent years, with applications spanning education, marketing, and remote work, as well as training for medicine, industry, and military. [1] All-day wearable XR displays are likely to reinvent computer interfaces in ways that rival the smartphone and personal computer, dramatically changing the way we interact with both the digital and physical worlds, as well as with other people. As we move toward a future in which seeing and hearing virtual objects is commonplace, we must also consider another important sensory aspect─touch.The sensation of touch is critical to our ability to interact with objects in the virtual world just as it is in the physical world, yet there remain significant challenges in synthesizing believable haptic interactions. The earliest haptic device designers proposed that for interactions to feel realistic, the haptic device must make free space feel free and must render stiff virtual objects. [2] These objectives led to the development of probe-based devices that exhibited low inertia and little to no backlash in their transmission mechanisms, and required anchoring to desktop surfaces so that world-grounded stiffnesses and resistance could be rendered to the user. XR haptic device designers are presented with yet more challenges. XR devices not only need to meet free space and stiffness criteria but must do so in an ungrounded, low encumbrance manner. So far, most efforts have focused on wireless haptic controllers, [3][4][5] fingertip displays, [6] and haptic gloves. [7] While these devices address the free-space consideration, they often cannot render virtual stiffnesses and, critically, prevent or degrade concurrent interaction with physical objects in all-day XR contexts. Soft, skin-like materials and devices may show promise in this way, imposing negligible physical burden on users, while delivering reliable sensations and sufficient forces to the skin [8][9][10] ; however, much of this technology is still under development and further from implementation.How can we render virtual stiffnesses without being grounded to the world or encumbering the hands? One method that has

show abstract

Section: Discussionmentioning

confidence: 99%

Multisensory Pseudo‐Haptics for Rendering Manual Interactions with Virtual Objects

Pezent

Macklin

Yau

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…The actuator is placed inside the stylus under the user's grip position and presents the vibrotactile sensation to the user's finger through the stylus's body. The stylus device is compatible with other data-driven haptic rendering methods for stylusshaped devices [47], [48], [49], [50], [51]. They synthesize vibrotactile patterns of various textures using prerecorded vibration patterns.…”

Section: Haptic Devices Controlled By Pvlcmentioning

confidence: 99%

HaptoMapping: Visuo-Haptic Augmented Reality by Embedding User-Imperceptible Tactile Display Control Signals in a Projected Image

Miyatake

Hiraki

Iwai

et al. 2023

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

This paper proposes HaptoMapping, a projection-based visuo-haptic augmented reality (VHAR) system, that can render visual and haptic content independently and present consistent visuo-haptic sensations on physical surfaces. HaptoMapping controls wearable haptic displays by embedded control signals that are imperceptible to the user in projected images using a pixel-level visible light communication technique. The prototype system is comprised of a high-speed projector and three types of haptic devices-finger worn, stylus, and arm mounted. The finger-worn and stylus devices present vibrotactile sensations to a user's fingertips. The arm-mounted device presents stroking sensations on a user's forearm using arrayed actuators with a synchronized hand projection mapping. We identified that the developed system's maximum latency of haptic from visual sensations was 93.4 ms. We conducted user studies on the latency perception of our VHAR system. The results revealed that the developed haptic devices can present haptic sensations without user-perceivable latencies, and the visual-haptic latency tolerance of our VHAR system was 100, 159, 500 ms for the finger-worn, stylus, and arm-mounted devices, respectively. Another user study with the arm-mounted device discovered that the visuo-haptic stroking system maintained both continuity and pleasantness when the spacing between each substrate was relatively sparse, such as 20 mm, and significantly improved both the continuity and pleasantness at 80 and 150 mm/s when compared to the haptic only stroking system. Lastly, we introduced four potential applications in daily scenes. Our system methodology allows for a wide range of VHAR application design without concern for latency and misalignment effects.

show abstract

“…Due to the recent breakthroughs on computer graphics research, the transition from traditional 2D visual content to adaptive 3D mixed reality worlds is straightforward and showing promising results 1 . In haptics, most of the existing works for vibrotactile feedback rendering or sound rendering use a tool-based texture interaction approach, where the user moves the tool over the virtual texture, and vibrotactile feedback is rendered via an actuator 2 – 4 , or sound is rendered through headphones 5 . Independently, both vibrotactile feedback rendering and sound rendering demonstrated sufficient reconstruction accuracy when applying the data-driven paradigm.…”

Section: Introductionmentioning

confidence: 99%

“…A parameter estimation algorithm is presented for sound synthesis in 12 , which overcomes the limitation of linear modal synthesis (i.e., frictional multibody contact formulation 10 ). However, the introduction of more sophisticated textures (i.e., anisotropic textures) has led to challenges with the intricate physical models and computational demands in real-time physics-based simulations of tool-surface interactions 5 , 17 , 18 . Although simplification attempts have been made but at the cost of reducing perceptual realism, impacting virtual experience authenticity.…”

Section: Introductionmentioning

confidence: 99%

A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering

Joolee,

Uddin

2024

Sci Rep

View full text Add to dashboard Cite

Data-driven approaches are often utilized to model and generate vibrotactile feedback and sounds for rigid stylus-based interaction. Nevertheless, in prior research, these two modalities were typically addressed separately due to challenges related to synchronization and design complexity. To this end, we introduce a novel multimodal multitask deep learning framework. In this paper, we developed a comprehensive end-to-end data-driven system that encompasses the capture of contact acceleration signals and sound data from various texture surfaces. This framework introduces novel encoder-decoder networks for modeling and rendering vibrotactile feedback through an actuator while routing sound to headphones. The proposed encoder-decoder networks incorporate stacked transformers with convolutional layers to capture both local variability and overall trends within the data. To the best of our knowledge, this is the first attempt to apply transformer-based data-driven approach for modeling and rendering of vibrotactile signals as well as sounds during tool-surface interactions. In numerical evaluations, the proposed framework demonstrates a lower RMS error compared to state-of-the-art models for both vibrotactile signals and sound data. Additionally, subjective similarity evaluation also confirm the superiority of proposed method over state-of-the-art.

show abstract

Towards Multisensory Perception: Modeling and Rendering Sounds of Tool-Surface Interactions

Cited by 10 publications

References 28 publications

Multisensory Pseudo‐Haptics for Rendering Manual Interactions with Virtual Objects

Multisensory Pseudo‐Haptics for Rendering Manual Interactions with Virtual Objects

HaptoMapping: Visuo-Haptic Augmented Reality by Embedding User-Imperceptible Tactile Display Control Signals in a Projected Image

A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering

Contact Info

Product

Resources

About