SeGAN: Segmenting and Generating the Invisible

Ehsani, Kiana; Mottaghi, Roozbeh; Farhadi, Ali

doi:10.1109/cvpr.2018.00643

Cited by 168 publications

(145 citation statements)

References 51 publications

Supporting

Mentioning

142

Contrasting

Unclassified

Order By: Relevance

“…Layered scene representations come in a diversity of contexts, such as depth ordering of semantic maps [48,39,16] and color images [7], motion analysis and optical flow [42,38], stereo reconstruction [3], scene decomposition in depth surfaces [28] and planes [27]. Our focus is on the Layered Depth Images (LDI) introduced by Shade et al [35], which refer to a single view representation of a scene that contains multiple layers of RGB-D information.…”

Section: Related Workmentioning

confidence: 99%

“…The advantage is that the 3D mesh captures all the available information in the scene, while an image-based approach [6] only captures information which is present in the set of consecutive image frames to be warped. For every frame, we render the visible instances separately, similarly to [7]. In addition to the color images and the visibility masks, we extract depth maps and object categories for every instance.…”

Section: Data Generationmentioning

confidence: 99%

See 1 more Smart Citation

Object-Driven Multi-Layer Scene Decomposition From a Single Image

Dhamo

Navab

Tombari

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We present a method that tackles the challenge of predicting color and depth behind the visible content of an image. Our approach aims at building up a Layered Depth Image (LDI) from a single RGB input, which is an efficient representation that arranges the scene in layers, including originally occluded regions. Unlike previous work, we enable an adaptive scheme for the number of layers and incorporate semantic encoding for better hallucination of partly occluded objects. Additionally, our approach is object-driven, which especially boosts the accuracy for the occluded intermediate objects. The framework consists of two steps. First, we individually complete each object in terms of color and depth, while estimating the scene layout. Second, we rebuild the scene based on the regressed layers and enforce the recomposed image to resemble the structure of the original input. The learned representation enables various applications, such as 3D photography and diminished reality, all from a single RGB image. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Data Generationmentioning

confidence: 99%

Object-Driven Multi-Layer Scene Decomposition From a Single Image

Dhamo

Navab

Tombari

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…This work is closely related to amodal segmentation works [12], [13], [14]. Generally, amodal perception refers to the intrinsic ability of humans to perceive objects as complete even if they are only partially visible.…”

Section: Introductionmentioning

confidence: 94%

Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions

Purkait

Zach

Reid

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Semantic segmentation and instance level segmentation made substantial progress in recent years due to the emergence of deep neural networks (DNNs). A number of deep architectures with Convolution Neural Networks (CNNs) were proposed that surpass the traditional machine learning approaches for segmentation by a large margin. These architectures predict the directly observable semantic category of each pixel by usually optimizing a cross entropy loss. In this work we push the limit of semantic segmentation towards predicting semantic labels of directly visible as well as occluded objects or objects parts, where the network's input is a single depth image. We group the semantic categories into one background and multiple foreground object groups, and we propose a modification of the standard cross-entropy loss to cope with the settings. In our experiments we demonstrate that a CNN trained by minimizing the proposed loss is able to predict semantic categories for visible and occluded object parts without requiring to increase the network size (compared to a standard segmentation task). The results are validated on a newly generated dataset (augmented from SUNCG) dataset.

show abstract

“…For instance, Radford A generated faces in various characteristics. Ehsani K completed the invisible section of scenes. There have already been some GAN‐based attempts on segmentation.…”

Section: Introductionmentioning

confidence: 99%

Accurate colorectal tumor segmentation for CT scans based on the label assignment generative adversarial network

et al. 2019

View full text Add to dashboard Cite

Purpose Colorectal tumor segmentation is an important step in the analysis and diagnosis of colorectal cancer. This task is a time consuming one since it is often performed manually by radiologists. This paper presents an automatic postprocessing module to refine the segmentation of deep networks. The label assignment generative adversarial network (LAGAN) is improved from the generative adversarial network (GAN) and assigns labels to the outputs of deep networks. We apply the LAGAN to segment colorectal tumors in computed tomography (CT) scans and explore the performances of different combinations of deep networks. Material and methods A total of 223 patients with colorectal cancer (CRC) are enrolled in the study. The CT scans of the colorectal tumors are first segmented by FCN32 and Unet separately, which output probabilistic maps. Then, the probabilistic maps are labeled by the LAGAN and finally, the binary segmentation results are obtained. The LAGAN consists of a generating model and a discriminating model. The generating model utilizes the probabilistic maps from deep networks to imitate the distribution of the ground truths, and the discriminating model attempts to distinguish generations and ground truths. Through competitive training, the generating model of the LAGAN can realize label assignments for the probabilistic maps. Results The LAGAN increases the DSC of FCN32 from 81.83% ± 0.35% to 90.82% ± 0.36%. In the Unet‐based segmentation, the LAGAN increases the DSC from 86.67% ± 0.70% to 91.54% ± 0.53%. It takes approximately 10 ms to refine a single CT slice. Conclusions The results demonstrate that the LAGAN is a robust and flexible module, which can be used to refine the segmentation of diverse deep networks. Compared with other networks, the LAGAN can achieve desirable segmented accuracy for colorectal tumors.

show abstract

SeGAN: Segmenting and Generating the Invisible

Cited by 168 publications

References 51 publications

Object-Driven Multi-Layer Scene Decomposition From a Single Image

Object-Driven Multi-Layer Scene Decomposition From a Single Image

Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions

Accurate colorectal tumor segmentation for CT scans based on the label assignment generative adversarial network

Contact Info

Product

Resources

About