Action Conditioned Tactile Prediction: case study on slip prediction

Abstract

Tactile predictive models can be useful across several robotic manipulation tasks, e.g. robotic pushing, robotic grasping, slip avoidance, and in-hand manipulation. However, available tactile prediction models are mostly studied for image-based tactile sensors and there is no comparison study indicating the best performing models. In this paper, we presented two novel data-driven action-conditioned models for predicting tactile signals during real-world physical robot interaction tasks (1) action condition tactile prediction and (2) action conditioned tactile-video prediction models. We use a magnetic-based tactile sensor that is challenging to analyse and test state-of-the-art predictive models and the only existing bespoke tactile prediction model. We compare the performance of these models with those of our proposed models. We perform the comparison study using our novel tactile enabled dataset containing 51,000 tactile frames of a real-world robotic manipulation task with 11 flat-surfaced household objects. Our experimental results demonstrate the superiority of our proposed tactile prediction models in terms of qualitative, quantitative and slip prediction scores.

Spectro-Temporal Recurrent Neural Network for Robotic Slip Detection with Piezoelectric Tactile Sensor

“…2) Learning models: Typical machine-learning classifiers as Support Vector Machine and Random Forest have already shown good results for slip detection, with generalization to novel objects [12] [13]. Deep learning models are also very common.…”

“…3) Incipient slip and slip prediction: Mandil et al proposed a slip-prediction system by classifying predicted future sensor signals [13]. This is a very complex problem and it requires taking future robot actions into account, as they influence tactile states.…”

Tactile-Sensing Technologies: Trends, Challenges and Outlook in Agri-Food Manipulation

“…In more recent research, the ability to push aside occluding unripe strawberries was explored in Nazari et al [29]; Figure 7. Occlusion from a robot's end effector causes physical interaction tasks, like pushing, to be a significant issue.Tactile sensation can be used to improve pushing performance and a robot's physical interaction perception Mandil and Ghalamzan-E [121]. Nazari et al [29] showed that pushing aside occluding strawberries can be performed with tactile sensation alone, enabling harvesting in these complex scenarios.…”

“…Additionally, tactile data can be employed to estimate the frictional properties between the object and the environment, which can be used to predict the object's motion and plan more effective pushing actions [179]. Using a combination of vision and tactile sensation was shown to produce more accurate object location predictions over extended time horizons [121]. Machine learning techniques have shown promise in enhancing objectpushing capabilities using tactile data.…”

“…Calandra et al [240] used a similar approach of visuo-tactile fusion by a CNN for grasp stability prediction. Mandil and Ghalamzan-E [121] proposed a multi-modal video prediction model by combining tactile and visual data in novel action-conditioned recurrent neural networks. A strong potential of visual and tactile fusion, which is not yet well explored, lies in the problem of fruit ripeness estimation, where the visual data capture the colour features for ripeness and tactile data measure the stiffness of the fruit.…”