Unseen Object Instance Segmentation for Robotic Environments

Xie, Christopher; Xiang, Yu; Mousavian, Arsalan; Fox, Dieter

doi:10.48550/arxiv.2007.08073

Cited by 10 publications

(12 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xiang et al [4] clusters object instances in 2 The respective config will be made publicly available. deep feature space to circumvent set prediction, and [69] merges a foreground mask with object center vectors which is then refined by RGB data. As an upper performance bound we train a MaskRCNN directly on photorealistically rendered 3D models of the YCB-V dataset used in the BOP Challenge [70].…”

Section: Comparison To the State-of-the-art Methodsmentioning

confidence: 99%

Unknown Object Segmentation from Stereo Images

Durner¹,

Boerdijk²,

Sundermeyer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although instance-aware perception is a key prerequisite for many autonomous robotic applications, most of the methods only partially solve the problem by focusing solely on known object categories. However, for robots interacting in dynamic and cluttered environments, this is not realistic and severely limits the range of potential applications. Therefore, we propose a novel object instance segmentation approach that does not require any semantic or geometric information of the objects beforehand. In contrast to existing works, we do not explicitly use depth data as input, but rely on the insight that slight viewpoint changes, which for example are provided by stereo image pairs, are often sufficient to determine object boundaries and thus to segment objects. Focusing on the versatility of stereo sensors, we employ a transformer-based architecture that maps directly from the pair of input images to the object instances. This has the major advantage that instead of a noisy, and potentially incomplete depth map as an input, on which the segmentation is computed, we use the original image pair to infer the object instances and a dense depth map. In experiments in several different application domains, we show that our Instance Stereo Transformer (INSTR) algorithm outperforms current state-of-the-art methods that are based on depth maps. Training code and pretrained models will be made available.

show abstract

Section: Comparison To the State-of-the-art Methodsmentioning

confidence: 99%

Unknown Object Segmentation from Stereo Images

Durner¹,

Boerdijk²,

Sundermeyer³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We compare three different segmentation approaches: UOIS-net-3D, geometric clustering, and a combined method. UOIS-net-3D [29] is a neural-network model that takes RGB-D images as input and returns a segmentation of the scene. It assumes that objects are generally resting on a table; it attempts to segment out image regions corresponding to the table, as well as a set of objects.…”

Section: A Segmentation Of Objects and Surfacesmentioning

confidence: 99%

Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

Curtis¹,

Fang²,

Kaelbling³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closedloop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable objects, their geometries, and their affordances. We explore several different ways of implementing such perceptual modules for segmentation, property detection, shape estimation, and grasp generation. We show how these modules are integrated within the PDDLStream task and motion planning framework. Finally, we demonstrate that this strategy can enable a single system to perform a wide variety of real-world multi-step manipulation tasks, generalizing over a broad class of objects, object arrangements, and goals, without any prior knowledge of the environment and without re-training.

show abstract

“…The problem with these clustering algorithms is that the results are sensitive to the hyperparameters such as number of segments and the neighborhood range. Other ideas for class-agnostic segmentation include using a network to predict object seeds [18] or searching over the space of candidate segmentations and returning one which scores well according to an "objectness" model [19]. However, the former method was only evaluated for the tabletop setting where most objects are roughly the same size whereas the latter method was only applied in outdoor environments where objects do not physically overlap with one another.…”

Section: A Class-agnostic Segmentationmentioning

confidence: 99%

LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Chen,

Kira,

Cho

2021

Preprint

View full text Add to dashboard Cite

3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.

show abstract

Unseen Object Instance Segmentation for Robotic Environments

Cited by 10 publications

References 83 publications

Unknown Object Segmentation from Stereo Images

Unknown Object Segmentation from Stereo Images

Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Contact Info

Product

Resources

About