Recurrent convolutional neural networks for object-class segmentation of RGB-D video

Pavel, Mircea Serban; Schulz, Hannes; Behnke, Sven

doi:10.1109/ijcnn.2015.7280820

Cited by 17 publications

(9 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Civera et al [27] used a monocular SLAM system to create 3D environment maps and then inserted the modeled object from the built database. Similarly, Pavel et al [28] also required priori 3D object models. Although these methods perform object-oriented semantic mapping, the requirement for priori knowledge of modeling objects makes it difficult for them to be applied in real-time human-robot interaction.…”

Section: Semantic Instance-awarementioning

confidence: 99%

“…Another project worth mentioning is [35]. Although it also combines a CNN and SLAM to generate 3D semantic mapping, it adds a recurrent neural network (RNN) [28] in data association.…”

Section: Semantic Instance-awarementioning

confidence: 99%

“…the rapid development of the convolutional neural network, semantic-related tasks in real-world environments have shown some remarkable results. Beginning with the object detection [3,28] in RGB images, soon afterwards, Mask R-CNN came out which is further able to predict a per-pixel semantically annotated mask for each of the detected instances, achieving state-of-the-art results on the COCO [36] instance-level semantic segmentation task. Other similar works that are worth to mention, including YOLO [5] and SSD [6], deliver an outstanding performance in terms of accurately segmenting instances.…”

Section: Instance Detection and Segmentation Nowadays Withmentioning

confidence: 99%

See 2 more Smart Citations

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Chen

et al. 2020

Discrete Dynamics in Nature and Society

View full text Add to dashboard Cite

Scene parsing plays a crucial role when accomplishing human-robot interaction tasks. As the “eye” of the robot, RGB-D camera is one of the most important components for collecting multiview images to construct instance-oriented 3D environment semantic maps, especially in unknown indoor scenes. Although there are plenty of studies developing accurate object-level mapping systems with different types of cameras, these methods either process the instance segmentation problem in completed mapping or suffer from a critical real-time issue due to heavy computation processing required. In this paper, we propose a novel method to incrementally build instance-oriented 3D semantic maps directly from images acquired by the RGB-D camera. To ensure an efficient reconstruction of 3D objects with semantic and instance IDs, the input RGB images are operated by a real-time deep-learned object detector. To obtain accurate point cloud cluster, we adopt the Gaussian mixture model as an optimizer after processing 2D to 3D projection. Next, we present a data association strategy to update class probabilities across the frames. Finally, a map integration strategy fuses information about their 3D shapes, locations, and instance IDs in a faster way. We evaluate our system on different indoor scenes including offices, bedrooms, and living rooms from the SceneNN dataset, and the results show that our method not only builds the instance-oriented semantic map efficiently but also enhances the accuracy of the individual instance in the scene.

show abstract

Section: Semantic Instance-awarementioning

confidence: 99%

“…Another project worth mentioning is [35]. Although it also combines a CNN and SLAM to generate 3D semantic mapping, it adds a recurrent neural network (RNN) [28] in data association.…”

Section: Semantic Instance-awarementioning

confidence: 99%

Section: Instance Detection and Segmentation Nowadays Withmentioning

confidence: 99%

See 1 more Smart Citation

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Chen

et al. 2020

Discrete Dynamics in Nature and Society

View full text Add to dashboard Cite

show abstract

“…It mainly focuses on semi-supervised approaches [1] [19] [20] that propagate the labels in one or more annotated frames to the entire video. In [17] a method that uses a combination of Recurrent Neural Networks (RNN) and CNN for RGB-D video segmentation is presented. However, their proposed architecture is difficult to train because of the vanishing gradient.…”

Section: Introductionmentioning

confidence: 99%

Convolutional gated recurrent networks for video segmentation

Siam

Valipour

Jägersand

et al. 2017

2017 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Semantic segmentation has recently witnessed major progress, where fully convolutional neural networks have shown to perform well. However, most of the previous work focused on improving single image segmentation. To our knowledge, no prior work has made use of temporal video information in a recurrent network. In this paper, we introduce a novel approach to implicitly utilize temporal data in videos for online semantic segmentation. The method relies on a fully convolutional network that is embedded into a gated recurrent architecture. This design receives a sequence of consecutive video frames and outputs the segmentation of the last frame. Convolutional gated recurrent networks are used for the recurrent part to preserve spatial connectivities in the image. Our proposed method can be applied in both online and batch segmentation. This architecture is tested for both binary and semantic video segmentation tasks. Experiments are conducted on the recent benchmarks in SegTrack V2, Davis, CityScapes, and Synthia. Using recurrent fully convolutional networks improved the baseline network performance in all of our experiments. Namely, 5% and 3% improvement of F-measure in Seg-Track2 and Davis respectively, 5.7% improvement in mean IoU in Synthia and 3.5% improvement in categorical mean IoU in CityScapes. The performance of the RFCN network depends on its baseline fully convolutional network. Thus RFCN architecture can be seen as a method to improve its baseline segmentation network by exploiting spatiotemporal information in videos.

show abstract

“…For example, Stückler et al [25] perform RGB-D SLAM to estimate the camera motion and aggregate semantic segmentations from multiple views in 3D. Pavel et al [26] directly train hierarchical recurrent convolutional neural networks on object class segmentation from RGB-D video. In this work, we do not address temporal integration and process only individual RGB-D frames.…”

Section: Introductionmentioning

confidence: 99%

Combining Semantic and Geometric Features for Object Class Segmentation of Indoor Scenes

Husain

Schulz

Dellen

et al. 2017

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Abstract-Scene understanding is a necessary prerequisite for robots acting autonomously in complex environments. Low-cost RGB-D cameras such as Microsoft Kinect enabled new methods for analyzing indoor scenes and are now ubiquitously used in indoor robotics. We investigate strategies for efficient pixelwise object class labeling of indoor scenes that combine both pretrained semantic features transferred from a large color image dataset and geometric features, computed relative to the room structures, including a novel distance-from-wall feature, which encodes the proximity of scene points to a detected major wall of the room. We evaluate our approach on the popular NYU v2 dataset. Several deep learning models are tested, which are designed to exploit different characteristics of the data. This includes feature learning with two different pooling sizes. Our results indicate that combining semantic and geometric features yields significantly improved results for the task of object class segmentation.

show abstract

Recurrent convolutional neural networks for object-class segmentation of RGB-D video

Cited by 17 publications

References 16 publications

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

Convolutional gated recurrent networks for video segmentation

Combining Semantic and Geometric Features for Object Class Segmentation of Indoor Scenes

Contact Info

Product

Resources

About