X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions

Danielczuk, Michael; Angelova, Anelia; Vanhoucke, Vincent; Goldberg, Ken

doi:10.1109/iros45743.2020.9340984

Cited by 31 publications

(42 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In previous work, we presented X-RAY (maXimize Reduction in support Area of occupancY distribution) [5] and LAX-RAY (Lateral Access X-RAY) [11], which introduce mechanical search policies that attempt to maximally reduce either support area or entropy of an estimated target occupancy distribution. The target occupancy distribution encodes target object likelihoods within an RGBD image observation, and is estimated using a deep neural network trained on a dataset of simulated depth images and target object segmentation masks with corresponding ground-truth occupancy distributions.…”

Section: Methodsmentioning

confidence: 99%

“…Xiao et al [23] and Price et al [19] attempt to model this distribution using particle filtering and shape completion approaches respectively. Danielczuk et al [5] explicitly learn the distribution in image space by generating a large dataset of simulated depth images and corresponding target object occupancy distributions. The target occupancy distributions represent all locations of the target object that would result in the rendered depth image.…”

Section: Related Work a Mechanical Searchmentioning

confidence: 99%

“…Based on the neural network architecture and training pipeline in Danielczuk et al [5], the SLAX-RAY perception system provides the search policy with information on where the target object may be hidden by computing a target occupancy distribution from a depth image. It also segments the scene into object instances using the input depth image.…”

Section: B Slax-ray Perception Systemmentioning

confidence: 99%

See 2 more Smart Citations

Mechanical Search on Shelves using a Novel "Bluction" Tool

Huang¹,

Danielczuk²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency. However, this efficiency comes at the cost of reduced visibility and accessibility. When looking from a side (lateral) view of a shelf, most objects will be fully occluded, resulting in a constrained lateralaccess mechanical search problem. To address this problem, we introduce: (1) a novel bluction tool, which combines a thin pushing blade and suction cup gripper, (2) an improved LAX-RAY simulation pipeline and perception model that combines ray-casting with 2D Minkowski sums to efficiently generate target occupancy distributions, and (3) a novel SLAX-RAY search policy, which optimally reduces target object distribution support area using the bluction tool. Experimental data from 2000 simulated shelf trials and 18 trials with a physical Fetch robot equipped with the bluction tool suggest that using suction grasping actions improves the success rate over the highest performing push-only policy by 26% in simulation and 67% in physical environments.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Work a Mechanical Searchmentioning

confidence: 99%

Section: B Slax-ray Perception Systemmentioning

confidence: 99%

See 1 more Smart Citation

Mechanical Search on Shelves using a Novel "Bluction" Tool

Huang¹,

Danielczuk²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other approaches deal with occluded objects that may be partially or completely invisible [13], [14], [15], [16], [17]. Often in such heavily cluttered scenes, such as a bin [18], [13], [7], [14] or a shelf [19], [20], the agent has to move other objects around to search for the target. While these approaches learn to search for a given target object, a more generalized version of this problem is to identify and locate all objects present in the scene.…”

Section: Related Workmentioning

confidence: 99%

“…Similarly, in our work we train an agent to rearrange with the objective of revealing hidden objects. However, in contrast to prior approaches in object search [18], [14], we do not require an auxiliary bin to discard occluding objects. This provides our agent two advantages: (i) in real world environments, the robot may not have access to additional space where it can discard objects, giving our approach a competitive edge; and, (ii) at the end of its sequence of interaction, our robot knows the location and identity of every object present in the scene and can retrieve multiple objects without any additional exploration.…”

Section: Related Workmentioning

confidence: 99%

Graph-based Cluttered Scene Generation and Interactive Exploration using Deep Reinforcement Learning

Kumar¹,

Essa²,

Ha³

2021

Preprint

View full text Add to dashboard Cite

We introduce a novel method to teach a robotic agent to interactively explore cluttered yet structured scenes, such as kitchen pantries and grocery shelves, by leveraging the physical plausibility of the scene. We propose a novel learning framework to train an effective scene exploration policy to discover hidden objects with minimal interactions. First, we define a novel scene grammar to represent structured clutter. Then we train a Graph Neural Network (GNN) based Scene Generation agent using deep reinforcement learning (deep RL), to manipulate this Scene Grammar to create a diverse set of stable scenes, each containing multiple hidden objects. Given such cluttered scenes, we then train a Scene Exploration agent, using deep RL, to uncover hidden objects by interactively rearranging the scene. We show that our learned agents hide and discover significantly more objects than the baselines. We present quantitative results that prove the generalization capabilities of our agents. We also demonstrate sim-to-real transfer by successfully deploying the learned policy on a real UR10 robot to explore real-world cluttered scenes. The supplemental video can be found at: https://www.youtube.com/watch?v=T2Jo7wwaXss.

show abstract