Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail Images

Abstract

The study of dexterous manipulation has provided important insights in humans sensorimotor control as well as inspiration for manipulation strategies in robotic hands. Previous work focused on experimental environment with restrictions. Here we describe a method using the deformation and color distribution of the fingernail and its surrounding skin, to estimate the fingertip forces, torques and contact surface curvatures for various objects, including the shape and material of the contact surfaces and the weight of the objects. The proposed method circumvents limitations associated with sensorized objects, gloves or fixed contact surface type. In addition, compared with previous single finger estimation in an experimental environment, we extend the approach to multiple finger force estimation, which can be used for applications such as human grasping analysis. Four algorithms are used, c.q., Gaussian process (GP), Convolutional Neural Networks (CNN), Neural Networks with Fast Dropout (NN-FD) and Recurrent Neural Networks with Fast Dropout (RNN-FD), to model a mapping from images to the corresponding labels. The results further show that the proposed method has high accuracy to predict force, torque and contact surface.

Visual Estimation of Fingertip Pressure on Diverse Surfaces using Easily Captured Data

“…A number of approaches have demonstrated that visual cues can be used to estimate hand pressure, including fingertip color changes [12,37,38], soft tissue deformation [25], and cast shadows [21,22,24]. In contrast to this prior work, our method uses an external camera to view the whole hand from a distance and deep learning to take advantage of multiple types of cues.…”

Image-Based Force Estimation in Medical Applications: A Review

“…In their 2008 paper [7], t system estimated finger forces with an accuracy of 5% is trained using a 3D textured mesh model of finger reconstructed from 2D images. They finally extended their research in [124] where the fingertip forces and torques, as well as the shape and material of the contact surface, were studied with four different predictors including a Gaussian Process regression (GP), CNN, fast dropout neural network (NN-FD), and RNN with fast dropout (RNN-FD). According to their report, GP and CNN show higher accuracy than NN-FD and RNN-FD.…”

PressureVision: Estimating Hand Pressure from a Single RGB Image

“…This effect is most visible at the fingertip and underneath the fingernail, where a whitening of the tissue is visible. Various techniques have been proposed to estimate fingertip force using optical sensors focused on this effect [9,38,39].…”