Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Abstract

When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions -push, suction, grasp -until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15,000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap. Code and supplementary material can be found at http://ai.stanford.edu/mech-search. * Authors have contributed equally and names are in alphabetical order.

Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations

“…In this approach, the object region was estimated by the density-based spatial clustering of applications with noise (DBSCAN) algorithm, and a depth difference image (DDI) that represents the depth difference between adjacent areas is defined. Different frameworks have been presented in achieving grasping object in clutter such as active affordance exploration framework which leverages the privileges of affordance map and the active exploration [129], integrating perception, action selection, and manipulation policies to address a version of the Mechanical Search problem [130], actor model with neural network that combines Gaussian mixture and normalizing flows [131], joint learning of instance and semantic segmentation for robotic pickand-place with heavy occlusions in clutter [132], and predicting the quality and the pose of grasp using U-Grasping fully convolutional neural network(UG-Net) based on pixel-wise using depth image [133].…”

The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints

“…An investigation into the multi-step retrieval of an occluded object called Mechanical Search [15] specifically states: "[The] performance gap [between our method and a human supervisor] suggests a number of open questions, such as: Can better perception algorithms improve performance? Can we formulate different sets of low level policies to increase the diversity of manipulation capability?…”

Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter

“…The occluding objects make it or impossible to grasp the desired object, requiring the robot to interact first with the unknown clutter to improve the target's graspability. Such situations appear frequently in domains such as home robotics or even logistic centers, and is considered an instance of the Mechanical Search problem [1].…”

“…Previous approaches proposed carefully-coded heuristics [1,2] or learned [3,4] sequences of actions that try to discover and retrieve the desired object. In both cases, the problem is simplified by choosing the action space to be a set of linear pushes parameterized as a point on the clutter and a direction to push, and a retracting motion after each action.…”