Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation

Liu, Guan-Horng; Siravuru, Avinash; Prabhakar, Sai; Veloso, Manuela; Kantor, George

doi:10.48550/arxiv.1705.10422

Cited by 12 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been several recent works that can perform inference or complete downstream task in the presence of missing modalities [23,24,[33][34][35]37]. However, they need to know what modality is missing at inference time.…”

Section: Introductionmentioning

confidence: 99%

Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

Lee¹,

Tan²,

Zhu³

et al. 2020

Preprint

View full text Add to dashboard Cite

Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans do this everyday, relying on touch and proprioceptive feedback in visually-challenging environments. However, robots might not always know when their sensors are corrupted, as even broken sensors can return valid values. In this work, we introduce the Crossmodal Compensation Model (CCM), which can detect corrupted sensor modalities and compensate for them. CMM is a representation model learned with self-supervision that leverages unimodal reconstruction loss for corruption detection. CCM then discards the corrupted modality and compensates for it with information from the remaining sensors. We show that CCM learns rich state representations that can be used for contact-rich manipulation policies, even when input modalities are corrupted in ways not seen during training time.

show abstract

Section: Introductionmentioning

confidence: 99%

Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

Lee¹,

Tan²,

Zhu³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…[21] adopts RGB and depth image to estimate the surface normal of the object. [6] [14] [22] [23] [24] [25] based on RGB image with depth or point cloud to predict grasping policy. [26] [27] [28] [29] fuses RGB and haptic data to train a grasping network.…”

Section: B Multimodal Perceptionmentioning

confidence: 99%

Deep Robotic Grasping Prediction with Hierarchical RGB-D Fusion

Song

Wen

Liu

et al. 2022

Int. J. Control Autom. Syst.

View full text Add to dashboard Cite

Robotic arm grasping is a fundamental operation in robotic control task goals. Most current methods for robotic grasping focus on RGB-D policy in the table surface scenario or 3D point cloud analysis and inference in the 3D space. Comparing to these methods, we propose a novel real-time multimodal hierarchical encoder-decoder neural network that fuses RGB and depth data to realize robotic humanoid grasping in 3D space with only partial observation. The quantification of raw depth data's uncertainty and depth estimation fusing RGB is considered. We develop a general labeling method to label ground-truth on common RGB-D datasets. We evaluate the effectiveness and performance of our method on a physical robot setup and our method achieves over 90% success rate in both table surface and 3D space scenarios. The video is available in https://youtu.be/_iRyLcfbTfg.

show abstract

“…Though state-dependent intermediate costs are not commonlyseen in supervised problems until recently [75], it has been used extensively in the context of deep reinforcement learning to guide or stabilize training, e.g. the auxiliary tasks and losses [76], [77].…”

Section: Training Dnn With Optimal Controlmentioning

confidence: 99%

Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Liu,

Theodorou

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited. In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control. By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory. When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem. In addition, we can reveal convergence and generalization properties by studying the stochastic dynamics of optimization algorithms. This viewpoint features a wide range of theoretical study from information bottleneck to statistical physics. It also provides a principled way for hyper-parameter tuning when optimal control theory is introduced. Our framework fits nicely with supervised learning and can be extended to other learning problems, such as Bayesian learning, adversarial training, and specific forms of meta learning, without efforts. The review aims to shed lights on the importance of dynamics and optimal control when developing deep learning theory.

show abstract

Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation

Cited by 12 publications

References 12 publications

Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors

Deep Robotic Grasping Prediction with Hierarchical RGB-D Fusion

Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Contact Info

Product

Resources

About