Haptic perception using optoelectronic robotic flesh for embodied artificially intelligent agents

Abstract

Flesh encodes a variety of haptic information including deformation, temperature, vibration, and damage stimuli using a multisensory array of mechanoreceptors distributed on the surface of the human body. Currently, soft sensors are capable of detecting some haptic stimuli, but whole-body multimodal perception at scales similar to a human adult (surface area ~17,000 square centimeters) is still a challenge in artificially intelligent agents due to the lack of encoding. This encoding is needed to reduce the wiring required to send the vast amount of information transmitted to the processor. We created a robotic flesh that could be further developed for use in these agents. This engineered flesh is an optical, elastomeric matrix “innervated” with stretchable lightguides that encodes haptic stimuli into light: temperature into wavelength due to thermochromic dyes and forces into intensity due to mechanical deformation. By exploiting the optical properties of the constitutive materials and using machine learning, we infer spatiotemporal, haptic information from light that is read by an image sensor. We demonstrate the capabilities of our system in various assemblies to estimate temperature, contact location, normal and shear force, gestures, and damage from temporal snapshots of light coming from the entire haptic sensor with errors <5%.

Machine Learning for Tactile Perception: Advancements, Challenges, and Opportunities

“…In addition to solving classification problems, SVM holds the ability to estimate a continuous‐valued multivariate function, [ 92 ] which enables it to deal with regression tasks, including vibration sensing, [ 93 ] and force estimation. [ 20 ] Barreiros et al [ 20 ] reported a soft, optical, robotic flesh that was able to encode haptic stimuli and recognize contact force, position and gesture. SVM was used to regress the force, as well as classify touch location and gesture.…”

“…Reproduced with permission. [ 20 ] Copyright 2022, AAAS. c) KNN‐based foam posture recognition for detecting bend and twists and predicting angle.…”

“…[ 172 ] Touch location can also be achieved using classification learning, with each class representing an area of contact positions. An example was proposed in the work of Barreiros et al, [ 20 ] which divided the sensor into square segments and classified the contact information into the related locations.…”

“…magnetic sensors. [12][13][14][15][16][17][18][19][20][21][22][23] These sensors have been widely applied in wearable electronic sensing systems, [24] bio-integrated sensing systems, [25] robot perception, [2,26,27] and other areas. Though traditional tactile signals are analog signals, a trend for generating spikes has been growing steeply in designing tactile sensors during recent years due to the favor of the spikes' power efficiency, short delay, and, more importantly, physiological basis.…”

“…Classification deals with discrete labels, for example, predicting which object that tactile sensor is touching, [44][45][46][47][48] detecting whether a slip happens. [49,50] Regression is defined as solving tasks with continuous labels, such as estimating contact force [20,51,52] and position. [20] When continuous values are thresholded or discretized, regression problems can be reformulated into classification tasks.…”