Six Degree-of-Freedom Haptic Simulation of a Stringed Musical Instrument for Triggering Sounds

Abstract

Six degree-of-freedom (DoF) haptic rendering of multi-region contacts between a moving hand avatar and varied-shaped components of a music instrument is fundamental to realizing interactive simulation of music playing. There are two aspects of computational challenges: first, some components have significantly small sizes in some dimensions, such as the strings on a seven-string plucked instrument (e.g., Guqin), which makes it challenging to avoid pop-through during multi-region contact scenarios. Second, deformable strings may produce high-frequency vibration, which requires simulating diversified and subtle force sensations when a hand interacts with strings in different ways. In this paper, we propose a constraint-based approach to haptic interaction and simulation between a moving hand avatar and various parts of a string instrument, using a cylinder model for the string that has a large length-radius ratio and a sphere-tree model for the other parts that have complex shapes. Collision response algorithms based on configuration-based optimization is adapted to solve for the contact configuration of the hand avatar interacting with thin strings without penetration. To simulate the deformation and vibration of a string, a cylindrical volume with variable diameters is defined with response to the interaction force applied by the operator. Experimental results have validated the stability and efficiency of the proposed approach. Subtle force feelings can be simulated to reflect varied interaction patterns, to differentiate collisions between the hand avatar with a static or vibrating string and the effects of various colliding forces and touch locations on the strings.

Configuration-Based Optimization for Virtual Hand Haptic Simulation

“…It is still an open research question how to quantitatively define none-penetration and stability. In existing studies of virtual hand interaction and haptic simulation, the evaluation of non-penetration and stability was assessed and reported qualitatively, such as the visual display of the interaction with virtual objects for the evaluation of non-penetration [21], [24], and the display of force curve for stability [34], [35]. Similarly, in this paper Fig.…”

“…In order to improve the computation of contact mechanics involving many contact points, Talvas et al [22] introduced volume contact constraints [23] and aggregated all contacts on each phalanx into a minimal set of constraints, and onehand (15 phalanges) grasping with about 80 contacts could attain about 25 Hz. To achieve interactive play with a virtual music instrument, Wang et al [24] achieved 6-DoF haptic rendering with 1 kHz update for the interactions between a hand with a fixed gesture and a stringed musical instrument by extending configuration-based optimization methods [24]- [26]. When applying such method to the natural hand-based interaction involving almost 20 DoFs and hundreds of contacts, the update rate is less than 20 Hz and abnormal situations may occur, such as malformed posture, joint angle jump and handobject penetrations depicted in Fig.…”

“…Considering the limited movement of four metacarpals other than that of the thumb, we model a virtual palm without the thumb metacarpal. Existing studies have shown the simplicity and efficiency of sphere-tree models [29] for haptic simulation [24], [25], and thus we adopt a sphere-tree model to represent the complex-shaped palm by Bradshaw's sphere-tree tool-kit 2 . Fig.…”

Survey on Hand-Based Haptic Interaction for Virtual Reality

“…Geometric models are used to represent the 3D shape of the virtual hand, and geometric constraints of the palm and multiple fingers. Each element of the virtual hand can be represented by using mainstream discrete models, e.g., mesh model [16]- [23], voxel model [24], [25], point-shell model [26], [27], and sphere model [28] [29]. The mesh model (such as Fig.…”

“…As shown in Fig. 6, when modeling a virtual hand by a sphere-tree model [28], low-hierarchy levels (such as level 1 and level 2) provide coarse detection, and the accuracy of collision detection grows higher with the level of hierarchy increases. Tong et al [29] used a sphere tree to model the palm and achieved over 1 kHz collision detection for the whole hand interaction with five cone-frustum fingers.…”