Joint Semantic and Motion Segmentation for Dynamic Scenes using Deep Convolutional Networks

Haque, Nazrul; Reddy, N. Dinesh; Krishna, K. Madhava

doi:10.5220/0006129200750085

Cited by 9 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A higher scene level understanding can be obtained by integrating semantics of the scene to get better correlation between objects in the scene and the depth and ego-motion estimates. This is similar to using semantic motion segmentation [17,18]. Architectural changes could also be leveraged to get a stronger coupling between depth and pose by having a single network predicting both pose and depth in order to allow the network to be able to learn representations that capture the complex relation between both camera motion and scene depth.…”

Section: Discussionmentioning

confidence: 99%

Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

Prasad

Das

Bhowmick

2018

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing

View full text Add to dashboard Cite

Deep approaches to predict monocular depth and ego-motion have grown in recent years due to their ability to produce dense depth from monocular images. The main idea behind them is to optimize the photometric consistency over image sequences by warping one view into another, similar to direct visual odometry methods. One major drawback is that these methods infer depth from a single view, which might not effectively capture the relation between pixels. Moreover, simply minimizing the photometric loss does not ensure proper pixel correspondences, which is a key factor for accurate depth and pose estimations.In contrast, we propose a 2-view depth network to infer the scene depth from consecutive frames, thereby learning inter-pixel relationships. To ensure better correspondences, thereby better geometric understanding, we propose incorporating epipolar constraints to make the learning more geometrically sound. We use the Essential matrix obtained using Nistér's Five Point Algorithm, to enforce meaningful geometric constraints, rather than using it as training labels. This allows us to use lesser no. of trainable parameters compared to state-of-the-art methods. The proposed method results in better depth images and pose estimates, which capture the scene structure and motion in a better way. Such a geometrically constrained learning performs successfully even in cases where simply minimizing the photometric error would fail.

show abstract

Section: Discussionmentioning

confidence: 99%

Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

Prasad

Das

Bhowmick

2018

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing

View full text Add to dashboard Cite

show abstract

“…The current method however only performs pixel level inferences. A higher scene level understanding can be obtained by integrating semantics to get better correlation between objects in the scene and depth & ego-motion estimates, similar to semantic motion segmentation [19,20].…”

Section: Discussionmentioning

confidence: 99%

SfMLearner++: Learning Monocular Depth & Ego-Motion Using Meaningful Geometric Constraints

Prasad¹,

Bhowmick²

2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Most geometric approaches to monocular Visual Odometry (VO) provide robust pose estimates, but sparse or semidense depth estimates. Off late, deep methods have shown good performance in generating dense depths and VO from monocular images by optimizing the photometric consistency between images. Despite being intuitive, a naive photometric loss does not ensure proper pixel correspondences between two views, which is the key factor for accurate depth and relative pose estimations. It is a well known fact that simply minimizing such an error is prone to failures.We propose a method using Epipolar constraints to make the learning more geometrically sound. We use the Essential matrix, obtained using Nistér's Five Point Algorithm, for enforcing meaningful geometric constraints on the loss, rather than using it as labels for training. Our method, although simplistic but more geometrically meaningful, using lesser number of parameters, gives a comparable performance to state-of-the-art methods which use complex losses and large networks showing the effectiveness of using epipolar constraints. Such a geometrically constrained learning method performs successfully even in cases where simply minimizing the photometric error would fail.

show abstract

“…The spectral clustering methods are effective at data clustering but cannot handle outliers and noise and often require post-processing. In recent years, there are some motion segmentation algorithms based on deep learning [24,25,26,27], which usually obtain more accurate segmentation results. However, the results of the deep-learning-based method depend strongly on the semantic segmentation.…”

Section: Introductionmentioning

confidence: 99%

Motion Segmentation Based on Model Selection in Permutation Space for RGB Sensors

Zhao

Qin

Luo

2019

Sensors

View full text Add to dashboard Cite

Motion segmentation is aimed at segmenting the feature point trajectories belonging to independently moving objects. Using the affine camera model, the motion segmentation problem can be viewed as a subspace clustering problem—clustering the data points drawn from a union of low-dimensional subspaces. In this paper, we propose a solution for motion segmentation that uses a multi-model fitting technique. We propose a data grouping method and a model selection strategy for obtaining more distinguishable data point permutation preferences, which significantly improves the clustering. We perform extensive testing on the Hopkins 155 dataset, and two real-world datasets. The experimental results illustrate that the proposed method can deal with incomplete trajectories and the perspective effect, comparing favorably with the current state of the art.

show abstract

Joint Semantic and Motion Segmentation for Dynamic Scenes using Deep Convolutional Networks

Cited by 9 publications

References 26 publications

Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

SfMLearner++: Learning Monocular Depth & Ego-Motion Using Meaningful Geometric Constraints

Motion Segmentation Based on Model Selection in Permutation Space for RGB Sensors

Contact Info

Product

Resources

About