TANDEM: Learning Joint Exploration and Decision Making With Tactile Sensors

Xu, Jingxi; Song, Shuran; Ciocarlie, Matei

doi:10.1109/lra.2022.3193466

Cited by 10 publications

(11 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While prior work has been done to complete geometry using depth alone [39,175], none of these works consider tactile information. More recent works have looked at informing decision making using partial observations from tactile information [182].…”

Section: Robotic Visual Shape Understandingmentioning

confidence: 99%

“…This adjustment not only would provide a visually interesting demonstration; it would also provide incremental information about where best to look next. This system could be further improved by utilizing a discriminator to determine whether there is enough information, such as the work in Tandem [182]. Additionally, newer point cloud based neural network architectures may be beneficial by reducing the memory requirements for the model which would improve resolution of the reconstruction.…”

Section: Learned Visual Navigation Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

Providing mobile robots with the ability to manipulate objects has, despite decades of research, remained a challenging problem. The problem is approachable in constrained environments where there is ample prior knowledge of the environment layout and manipulatable objects. The challenge is in building systems that scale beyond specific situational instances and gracefully operate in novel conditions. In the past, researchers used heuristic and simple rule-based strategies to accomplish tasks such as scene segmentation or reasoning about occlusion. These heuristic strategies work in constrained environments where a roboticist can make simplifying assumptions about everything from the geometries of the objects to be interacted with, level of clutter, camera position, lighting, and a myriad of other relevant variables. The work in this thesis will demonstrate how to build a system for robotic mobile manipulation that is robust to changes in these variables. This robustness will be enabled by recent simultaneous advances in the fields of big data, deep learning, and simulation. The ability of simulators to create realistic sensory data enables the generation of massive corpora of labeled training data for various grasping and navigation-based tasks. It is now possible to build systems that work in the real world trained using deep learning entirely on synthetic data. The ability to train and test on synthetic data allows for quick iterative development of new perception, planning and grasp execution algorithms that work in many environments.To build a robust system, this thesis introduces a novel multiple-view shape reconstruction architecture that leverages unregistered views of the object. To navigate to objects without localizing the agent, this thesis introduces a novel panoramic target goal architecture that takes previous views of the agent to inform a policy to navigate through an environment. Additionally, a novel next-best-view methodology is introduced to allow the agent to move around the object and refine its initial understanding of the object. The results show that this deep learned sim-to-real approach performs best when compared to heuristic-based methods in terms of reconstruction quality and success-weighted-by-path-length (SPL). This approach is also adaptable to the environment and robot chosen due to its modular design.I would like to acknowledge all members of the Columbia Robotics Lab (CRLab) for making my stay there an enjoyable and rewarding experience. In particular, I would like to thank

show abstract

Section: Robotic Visual Shape Understandingmentioning

confidence: 99%

Section: Learned Visual Navigation Future Workmentioning

confidence: 99%

Learning Mobile Manipulation

Watkins¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While there are many works on tactile recognition of 2D objects [1], [4], [5], [6], [7], tactile recognition of 3D objects is less addressed in the community. [8] uses power grasps to collect tactile data and then train a deep neural network for recognition.…”

Section: A 3d Object Recognition With Tactile Sensorsmentioning

confidence: 99%

“…In contrast, learning-based exploration policies can be trained in an unsupervised fashion through trial and error. Noise can be incorporated into the training process so that the policy can be more robust [1]. However, [1] only handles 2D objects and their method does not scale directly to 3D objects.…”

Section: B Tactile Exploration Policymentioning

confidence: 99%

See 1 more Smart Citation

TANDEM3D: Active Tactile Exploration for 3D Object Recognition

Xu¹,

Lin²,

Song³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Tactile recognition of 3D objects remains a challenging task. Compared to 2D shapes, the complex geometry of 3D surfaces requires richer tactile signals, more dexterous actions, and more advanced encoding techniques. In this work, we propose TANDEM3D, a method that applies a co-training framework for exploration and decision making to 3D object recognition with tactile signals. Starting with our previous work, which introduced a co-training paradigm for 2D recognition problems, we introduce a number of advances that enable us to scale up to 3D. TANDEM3D is based on a novel encoder that builds 3D object representation from contact positions and normals using PointNet++. Furthermore, by enabling 6DOF movement, TANDEM3D explores and collects discriminative touch information with high efficiency. Our method is trained entirely in simulation and validated with real-world experiments. Compared to state-of-the-art baselines, TANDEM3D achieves higher accuracy and a lower number of actions in recognizing 3D objects and is also shown to be more robust to different types and amounts of sensor noise.

show abstract

Skin‐Inspired Thermosensitive Tactile Sensor Based on Thermally Conductive and Viscous Interface Composites for Rocks

et al. 2023

View full text Add to dashboard Cite

Achieving high thermal conductivity and exceptional interfacial adhesion simultaneously in thermosensitive tactile recognition sensors poses a significant challenge. A copolymer, poly([[(butylamino)carbonyl]oxy]ethyl‐ester)‐co‐polydimethylsiloxane (referred to as PP), is synthesized and subsequently complexed with alumina particles coated with liquid metal (LMAl2O3) to prepare a composite material called PP/LMAl2O3 with high thermal conductivity and strong interfacial adhesion to address this challenge. The best thermal conductivity (4.43 W m−1 K−1), electrical insulation (10−6–10−7 S m−1), and adhesion properties derived from hydrogen bonding (1316 N m−2) are obtained by adjusting the volume fraction of PP and LMAl2O3 in PP/LMAl2O3. PP/LMAl2O3 with high thermal conductivity and high interface adhesion can efficiently transfer heat between thermal flux sensors and the objects being sensed, reliably detecting small thermal flux variations and ensuring accurate thermal flux measurements. In this study, PP/LMAl2O3 is used to make up thermosensitive tactile sensor. Surprisingly, PP/LMAl2O3 demonstrates high thermal signal sensitivity for tactile recognition applications, allowing the smart thermosensitive tactile sensor system to distinguish unknown rock materials even in the dark. Overall, PP/LMAl2O3 may function as a fundamental material in thermosensitive tactile sensors for lithology identification.

show abstract

TANDEM: Learning Joint Exploration and Decision Making With Tactile Sensors

Cited by 10 publications

References 27 publications

Learning Mobile Manipulation

Learning Mobile Manipulation

TANDEM3D: Active Tactile Exploration for 3D Object Recognition

Skin‐Inspired Thermosensitive Tactile Sensor Based on Thermally Conductive and Viscous Interface Composites for Rocks

Contact Info

Product

Resources

About