Towards safe human-to-robot handovers of unknown containers

Pang, Yik Lung; Xompero, Alessio; Oh, Chang-Hwan; Cavallaro, Andrea

doi:10.1109/ro-man50785.2021.9515350

Cited by 17 publications

(19 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The goal is to estimate the width at the top ( wj t , in mm) and at the bottom ( wj b , in mm), and height ( hj , in mm) of the container. Algorithms designed for the challenge are expected to estimate these physical properties to compute the mass of the filling as [11] to evaluate the impact of estimation errors. Given video frames and audio signals from the CORSMAL Containers Manipulation (CCM) dataset [8,12]), the results of T1 (filling level), T2 (filling type), and T3 (container capacity) are used to compute the filling mass, which is added to T4 (container mass) for estimating the mass of the object (container + filling).…”

Section: The Tasksmentioning

confidence: 99%

“…The estimated dimensions (T5) are used to visualise the container. The simulator also uses object annotations, such as 6D poses over time, the true weight (container + filling), a 3D mesh model reconstructed offline with a vision baseline [11], and the frame where the object is ready to be grasped by the simulated robot arm, for performing and visualising the handover.…”

Section: The Tasksmentioning

confidence: 99%

“…The challenge adopts a real-to-simulation framework [11] that complements the CCM dataset with a human-to-robot handover in the PyBullet simulation environment [13]. The framework uses the physical properties of a manipulated container estimated by a perception algorithm.…”

Section: Real-to-sim Visualisationmentioning

confidence: 99%

“…The simulator renders a 3D object model reconstructed offline by a vision baseline in manually selected frames with no occlusions [11]. The weight of the object used by the simulator is the true, annotated value.…”

Section: Real-to-sim Visualisationmentioning

confidence: 99%

“…We carefully defined a set of performance scores to directly evaluate and systematically compare the algorithms on each task. Moreover, to assess the accuracy of the estimations and visualise the safeness of human-to-robot handovers, we implemented a real-to-simulation framework [11] that provides indirect high-level evaluations on the impact of these tasks (see Fig. 2).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Audio-Visual Object Classification for Human-Robot Collaboration

Xompero¹,

Pang²,

Patten³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human-robot collaboration requires the contactless estimation of the physical properties of containers manipulated by a person, for example while pouring content in a cup or moving a food box. Acoustic and visual signals can be used to estimate the physical properties of such objects, which may vary substantially in shape, material and size, and also be occluded by the hands of the person. To facilitate comparisons and stimulate progress in solving this problem, we present the CORSMAL challenge and a dataset to assess the performance of the algorithms through a set of well-defined performance scores. The tasks of the challenge are the estimation of the mass, capacity, and dimensions of the object (container), and the classification of the type and amount of its content. A novel feature of the challenge is our real-to-simulation framework for visualising and assessing the impact of estimation errors in human-to-robot handovers.

show abstract

Section: The Tasksmentioning

confidence: 99%

Section: The Tasksmentioning

confidence: 99%

Section: Real-to-sim Visualisationmentioning

confidence: 99%

Section: Real-to-sim Visualisationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Audio-Visual Object Classification for Human-Robot Collaboration

Xompero¹,

Pang²,

Patten³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Learning Human-to-Robot Handovers from Point Clouds

Christen,

Yang,

Pérez-D'Arpino

et al. 2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Vision-based human-to-robot handover is an important and challenging task in human-robot interaction. Recent work has attempted to train robot policies by interacting with dynamic virtual humans in simulated environments, where the policies can later be transferred to the real world. However, a major bottleneck is the reliance on human motion capture data, which is expensive to acquire and difficult to scale to arbitrary objects and human grasping motions. In this paper, we introduce a framework that can generate plausible human grasping motions suitable for training the robot. To achieve this, we propose a hand-object synthesis method that is designed to generate handover-friendly motions similar to humans. This allows us to generate synthetic training and testing data with 100x more objects than previous work. In our experiments, we show that our method trained purely with synthetic data is competitive with state-of-the-art methods that rely on real human motion data both in simulation and on a real system. In addition, we can perform evaluations on a larger scale compared to prior work. With our newly introduced test set, we show that our model can better scale to a large variety of unseen objects and human motions compared to the baselines.

show abstract