Tactile rendering based on skin stress optimization

Abstract

We present a method to render virtual touch, such that the stimulus produced by a tactile device on a user's skin matches the stimulus computed in a virtual environment simulation. To achieve this, we solve the inverse mapping from skin stimulus to device configuration thanks to a novel optimization algorithm. Within this algorithm, we use a device-skin simulation model to estimate rendered stimuli, we account for trajectory-dependent effects efficiently by decoupling the computation of the friction state from the optimization of device configuration, and we accelerate computations using a neural-network approximation of the device-skin model. Altogether, we enable real-time tactile rendering of rich interactions including smooth rolling, but also contact with edges, or frictional stick-slip motion. We validate our algorithm both qualitatively through user experiments, and quantitatively on a BioTac biomimetic finger sensor.

Survey on Hand-Based Haptic Interaction for Virtual Reality

“…For these two types of haptic devices, haptic stimulation is limited to the passive outer layer of the hand skin. Different from kinesthetic rendering, tactile rendering is therefore not subject to stability problems due to latency [102]. For such tactile rendering, the update rate does not affect stability as in kinesthetic rendering, but rather affects the bandwidth of the interactions that can be rendered.…”

“…The work [108] implemented finger interaction instead of full-hand interaction for computational efficiency. Later, Perez et al [103] [14] Skin: linear co-rotational FEM with strain-limiting constraints LCSM: SLSQP sequential quadratic programming routine in NLopt 3-DoF wearable thimble [95], open-chain and parallel mechanisms (1) Scenarios: exploring virtual surfaces with various properties (flat surface and edge) (2) Update rate: average 50 Hz (16ms for contact simulation, <1ms for unconstrained and 4ms for constrained device optimization) (3) Fidelity: outperforming plane-fitting and unconstrained optimization in force error Verschoor et al 2020 [102] Soft hand in [56] with a data-driven skin model (distal phalanx of the index finger) Skin stress optimization: nonlinear least-squares optimization by an active-set Gauss-Newton method 3-DoF wearable thimble [95], parallel mechanism (1) Scenarios: evaluating the ability to render diverse virtual interactions, including smooth rolling, contact with edges, and frictional stick-slip motion (2) Update rate: 80 Hz (optimization plus friction update) (3) Fidelity: the stimuli produced by the device match closely the recorded target stimuli Hirota et al 2022 [110] A finger model (the tetrahedral mesh is generated by the TetMesh program [111]) from BodyParts3d [112] Deformation matching: minimizing the error between the target skin deformation and the skin deformation caused by the device 128-pin finger-mounted tactile device [113] (1) Scenarios: comparing deformation matching with force matching in terms of the perception of the friction coefficient and of the perception of the friction direction (2) Update rate: non-real-time and driving the device with offline data (3) Fidelity: deformation matching performs better on the perception of the friction coefficient and friction direction than force matching A non-rigid mesh model driven by four depth cameras [123] Solving a minimization problem to optimize the sound field amplitude using Levenberg-Marquardt method Ultrasound tactile display proposed by Inoue et al [124] (1) Scenarios: reproducing a randomly determined amplitude distribution, and controlling the scattered sound field for a hand with a certain grasping shape (2) Update rate: the computation time is 1.784 s on CPU (Core i7-10700K CPU 3.80 GHz) and 0.4793 s on GPU (NVIDIA GeForce RTX 3090) (3) Fidelity: this work reproduces the amplitude distribution of the sound field with higher accuracy than conventional algorithms the nonlinear constraints on each Jacobi iteration and performs line-search optimization per iteration to guarantee a steady error reduction. This solver achieved more than 10× over previous approaches, and enabled multi-finger manipulation of a virtual object by adopting the contact surface matching tactile rendering algorithm [108].…”

Force-Aware Interface via Electromyography for Natural VR/AR Interaction

“…While users apply forces to a virtual object, they also receive haptic feedback from the object's response [Dangxiao et al 2019;Gonzalez et al 2021;Yoshida et al 2020]. For the latter, which has been addressed in computer graphics as haptic rendering [Lin and Otaduy 2008], researchers have explored various ways of applying tactile effects to users, ranging from grasping and touching [Choi et al 2016[Choi et al , 2018Verschoor et al 2020] to texture [Benko et al 2016], shear [Whitmire et al 2018] and gravity [Choi et al 2017].…”

Learning contact corrections for handle-based subspace dynamics

“…Friction is another source of trajectory-dependent deformations. Friction could be handled in a way similar to dynamics, e.g., by introducing previous states of the collider to the learning architecture, or also by modeling the friction state as an input explicitly [Verschoor et al 2020].…”