Learning Human-to-Robot Handovers from Point Clouds

Abstract

Vision-based human-to-robot handover is an important and challenging task in human-robot interaction. Recent work has attempted to train robot policies by interacting with dynamic virtual humans in simulated environments, where the policies can later be transferred to the real world. However, a major bottleneck is the reliance on human motion capture data, which is expensive to acquire and difficult to scale to arbitrary objects and human grasping motions. In this paper, we introduce a framework that can generate plausible human grasping motions suitable for training the robot. To achieve this, we propose a hand-object synthesis method that is designed to generate handover-friendly motions similar to humans. This allows us to generate synthetic training and testing data with 100x more objects than previous work. In our experiments, we show that our method trained purely with synthetic data is competitive with state-of-the-art methods that rely on real human motion data both in simulation and on a real system. In addition, we can perform evaluations on a larger scale compared to prior work. With our newly introduced test set, we show that our model can better scale to a large variety of unseen objects and human motions compared to the baselines.

MoVEInt: Mixture of Variational Experts for Learning Human–Robot Interactions From Demonstrations

“…Some extensions for extrapolating to such tasks can be explored by using better Generative models [8], [10], [42] or adding explicit constraints for combining reactive motion generation and planning [43]. Our future work would also explore incorporating task-related constraints such as object information for handover grasps [44], [45], force information for enabling natural interactive behaviors [46]- [48], and ergonomic and safety constraints for a more user-friendly interaction [49], [50].…”