Occlusion-Aware Search for Object Retrieval in Clutter

Abstract

We address the manipulation task of retrieving a target object from a cluttered shelf. When the target object is hidden, the robot must search through the clutter for retrieving it. Solving this task requires reasoning over the likely locations of the target object. It also requires physics reasoning over multi-object interactions and future occlusions. In this work, we present a data-driven hybrid planner for generating occlusionaware actions in closed-loop. The hybrid planner explores likely locations of the occluded target object as predicted by a learned distribution from the observation stream. The search is guided by a heuristic trained with reinforcement learning to act on observations with occlusions. We evaluate our approach in different simulation and real-world settings (video available on https://youtu.be/dY7YQ3LUVQg). The results validate that our approach can search and retrieve a target object in near real time in the real world while only being trained in simulation.

Safe, Occlusion-Aware Manipulation for Online Object Reconstruction in Confined Spaces

“…When object models are unknown, objects may need to be reconstructed [21,14]. When the number of hidden objects is given, their poses can be sampled inside the occlusion region [2,32,19]. Otherwise, the scene needs to be fully revealed when it is unknown in the worst case.…”

“…Camera Mounting When the camera is fixed on the wrist of the robot, the robot needs to move the camera to reveal more space in the scene [2]. On the other hand, when the camera is static in the scene, the robot needs to bring objects in front of it for sensing, thus requiring more complicated actions.…”

“…Model-Based Object Retrieval in a Shelf Many works in the literature focus on the problem of object search and retrieval with known object models from a shelf [10,23,2,32,19,15]. In this setting, The perception module is assumed to fully recognize the object once a sufficient part is observed, thus ignoring partial occlusion.…”

“…More recently, the focus has shifted to cases where object models are unavailable. Nevertheless, most of these methods do not provide theoretical guarantees, either due to the dependence on machine learning [2,8,18,15], or assumptions that are valid only in limited scenarios [14]. Learning-based methods bring great promise to achieve efficiency but are highly desirable if they can be applied within a safe and complete framework.…”

Robust Physics-Based Manipulation by Interleaving Open and Closed-Loop Execution

“…Li et al [36] propose push-net, a deep recurrent neural network that uses only an image as input to push objects of unknown physical properties. Bejjani et al [37] address the problem of occlusion in lateral access into shelves, with a hybrid planner based on a learned heuristic. Kiatos and Malassiotis [38] learn an optimal push policy to singulate objects in clutter with lateral pushing actions.…”