iModel: Interactive Co-segmentation for Object of Interest 3D Modeling

Kowdle, Adarsh; Batra, Dhruv; Chen, Wenchao; Chen, Tsuhan

doi:10.1007/978-3-642-35740-4_17

Cited by 18 publications

(15 citation statements)

References 48 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also use two standard datasets that have been used in prior automatic works [85]. We make all the datasets used in our works [53,55,56] publicly available 8 .…”

Section: Datasetsmentioning

confidence: 99%

Putting the User in the Loop for Image-Based Modeling

et al. 2014

Self Cite

View full text Add to dashboard Cite

Image-based modeling, the task of recovering the 3D structure of an object or a scene using 2D images is one of the primary goals of computer vision. However, 3D reconstruction in practice is a hard task due to the numerous scene irregularities. When a scene is captured from multiple viewpoints, irregularities such as textureless surfaces, specularities, thin structures, etc., make the reconstruction inaccurate. In addition, image sequences or videos of a scene captured in the wild often consist of dynamic or moving objects such as people, which makes the task of image-based modeling extremely hard. In this thesis, our goal is to tackle these problems to obtain a more accurate reconstruction of the scene. In particular, using the intuition that humans easily discern the 3D structure behind the photons, this thesis addresses these problems by putting the user in the loop with the image-based-modeling algorithm.In the first part of the thesis, we focus on image-based modeling of static scenes. We explore putting the user in the loop interacting with the algorithm by providing constraints via simple interactions on the image data, to help overcome the ill-effects of the scene irregularities. We introduce two algorithms within this framework. First, an algorithm where the user drives the process of image-based modeling. Second, a novel active-learning algorithm, which initiates the process of image-based modeling via an unsupervised algorithm and then guides the user to provide constraints when and where needed.In the second part of the thesis, we focus on image-based modeling of dynamic scenes captured by either a dynamic camera or a static camera. We develop algorithms that leverage the dynamic objects in the scene to aid the imagebased modeling algorithm. We propose novel algorithms that use the sparse, yet strong occlusion cues between the moving object and the scene, along with a realistic motion prior for the moving object to recover the depth of the scene.We explore putting the user in the loop within the framework, guided by the algorithm to provide useful depth-order constraints and show that we further improve the solution of the unsupervised algorithm.Finally, we present an end user application: iModel, which puts the user in the loop for object of interest 3D modeling on a mobile device and 3D printing of an object of interest. THESIS COMMITTEE

show abstract

“…We also use two standard datasets that have been used in prior automatic works [85]. We make all the datasets used in our works [53,55,56] publicly available 8 .…”

Section: Datasetsmentioning

confidence: 99%

Putting the User in the Loop for Image-Based Modeling

et al. 2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…At this stage, the algorithm knows which surface indicated by the user corresponds to the nonplanar object. We treat the scribbles corresponding to the nonplanar object as foreground scribbles and all other scribbles as background scribbles and use ideas from prior work by Kowdle et al [12] to obtain a 3D visual hull of the non-planar object via a 2-class co-segmentation, which is rendered using an independent mesh. The scene co-segmentation also allows us to create a composite texture map for the scene covering up holes due to occlusions as shown in Fig.3(a).…”

Section: Rendering Non-planar Objectsmentioning

confidence: 99%

“…, GMM p }. Specifically, we use colour features extracted from superpixels [12] on the labeled sites and fit GMMs for the corresponding classes. The data terms for all sites are then defined as the negative log-likelihood of the features given the class model.…”

Section: Scribbles To Scene Segmentationmentioning

confidence: 99%

“…They were first used by Boykov et al for interactive segmentation to indicate foreground and background [10]. Batra et al used scribbles for interactive cosegmentation [11], which was applied to object of interest 3D modeling by Kowdle et al [12]. Srivastava et al improve the 3D model obtained using their Make3D algorithm by using scribbles to enforce coplanarity in their MRF formulation [13].…”

Section: Introductionmentioning

confidence: 99%

“…The scene co-segmentation helps obtain a composite texture map for the scene eliminating holes due to occluding objects, giving a pleasing planar reconstruction of the scene. In addition to this, we use the co-segmentation of non-planar objects in the scene to obtain a visual hull for the occluding object [12], which is rendered as part of the prior planar reconstruction of the scene. We now describe our approach in detail.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Scribble based interactive 3D reconstruction via scene co-segmentation

Kowdle

Chang

Batra

et al. 2011

2011 18th IEEE International Conference on Image Processing

Self Cite

View full text Add to dashboard Cite

In this paper, we present a novel interactive 3D reconstruction algorithm which renders a planar reconstruction of the scene. We consider a scenario where the user has taken a few images of a scene from multiple poses. The goal is to obtain a dense and visually pleasing reconstruction of the scene, including non-planar objects. Using simple user interactions in the form of scribbles indicating the surfaces in the scene, we develop an idea of 3D scribbles to propagate scene geometry across multiple views and perform co-segmentation of all the images into the different surfaces and non-planar objects in the scene. We show that this allows us to render a complete and pleasing reconstruction of the scene along with a volumetric rendering of the non-planar objects. We demonstrate the effectiveness of our algorithm on both outdoor and indoor scenes including the ability to handle featureless surfaces.Index Terms-image based modeling, interactive 3D reconstruction INTRODUCTIONWe consider a scenario where the user has taken a few images of a scene from varied poses. The goal is to obtain a visually pleasing reconstruction of the scene. Automatic algorithms such as [1,2,3], etc have been shown to work well with a large collection of images. When the number of input images is restricted, these automatic algorithms fail to produce a dense plausible reconstruction. There are a number of multiview stereo algorithms which try to obtain a dense depth map for the scene from a set of images [4]. However, multiview stereo algorithms are known to be slow and with a small set of images the reconstruction is usually incomplete, leaving holes on textureless surfaces and specular reflections. In order to improve the reconstruction, some algorithms make planar approximations to the scene [5,6]. This allows for more visually pleasing reconstructions. However, these algorithms use features such as strong edges and lines which may be absent in textureless surfaces or non-planar objects (like walls, trees, people, etc). This has led to interactive algorithms. Prior interactive reconstruction algorithms, require involved user-interactions ranging from providing feature correspondence, to marking edges, plane boundaries and detailed line models of the scene [7,8,9]. In this paper, we present a novel scribble based interactive 3D reconstruction algorithm where we relax the interactions to mere scribbles and render a planar reconstruction of the scene. In a typical scene, nonplanar objects occluding the scene can result in holes in the scene reconstruction. The strength of our approach is the ability to use surface-level correspondence across multiple views to create composite texture maps for the scene thereby rendering pleasing planar reconstructions of the scene. Moreover, we use this correspondence to obtain volumetric reconstructions of the occluding object. All of this involves very simple interactions in the form of scribbles.Scribbles have been used in interactive algorithms in the past. They were first used by Boykov et al. fo...

show abstract

Deep Object Co-segmentation

Jafari

Rother

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This work presents a deep object co-segmentation (DOCS) approach for segmenting common objects of the same class within a pair of images. This means that the method learns to ignore common, or uncommon, background stuff and focuses on common objects. If multiple object classes are presented in the image pair, they are jointly extracted as foreground. To address this task, we propose a CNN-based Siamese encoder-decoder architecture. The encoder extracts high-level semantic features of the foreground objects, a mutual correlation layer detects the common objects, and finally, the decoder generates the output foreground masks for each image. To train our model, we compile a large object co-segmentation dataset consisting of image pairs from the PAS-CAL dataset with common objects masks. We evaluate our approach on commonly used datasets for co-segmentation tasks and observe that our approach consistently outperforms competing methods, for both seen and unseen object classes.

show abstract

iModel: Interactive Co-segmentation for Object of Interest 3D Modeling

Cited by 18 publications

References 48 publications

Putting the User in the Loop for Image-Based Modeling

Putting the User in the Loop for Image-Based Modeling

Scribble based interactive 3D reconstruction via scene co-segmentation

Deep Object Co-segmentation

Contact Info

Product

Resources

About