Keep it Unreal: Bridging the Realism Gap for 2.5D Recognition with Geometry Priors Only

Zakharov, Sergey; Planche, Benjamin; Wu, Ziyan; Hutter, Andreas; Kosch, Harald; Ilić, Slobodan

doi:10.1109/3dv.2018.00012

Cited by 29 publications

(22 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the eye training dataset RGB images are in different sensor space compared to the IR eye cameras inside the HMD, the detector might have a greater performance if trained with images that match the noise and spectral characteristics of the captured images. To decrease the domain gap between training data and application images, generative adversarial networks (GAN) have been proposed which learn to map between synthetic and real images with Rad et al [Rad et al 2019], Mueller et al [2018], and Zakharov et al [Zakharov et al 2018]. This approach could be used in our solution but it is necessary to have a different dataset to train the GAN network to learn the mapping between RBG and infrared images.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

Brito

Mitchell

2019

Proceedings of the 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry

View full text Add to dashboard Cite

Preparing datasets for use in the training of real-time face tracking algorithms for HMDs is costly. Manually annotated facial landmarks are accessible for regular photography datasets, but introspectively mounted cameras for VR face tracking have incompatible requirements with these existing datasets. Such requirements include operating ergonomically at close range with wide angle lenses, low-latency short exposures, and near infrared sensors. In order to train a suitable face solver without the costs of producing new training data, we automatically repurpose an existing landmark dataset to these specialist HMD camera intrinsics with a radial warp reprojection. Our method separates training into local regions of the source photos, i.e., mouth and eyes for more accurate local correspondence to the mounted camera locations underneath and inside the fully functioning HMD. We combine per-camera solved landmarks to yield a live animated avatar driven from the user's face expressions. Critical robustness is achieved with measures for mouth region segmentation, blink detection and pupil tracking. We quantify results against the unprocessed training dataset and provide empirical comparisons with commercial face trackers. CCS CONCEPTS • Computing methodologies → Motion capture; VR.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

Brito

Mitchell

2019

Proceedings of the 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry

View full text Add to dashboard Cite

show abstract

“…The model was trained on UOAIS-Sim dataset following the standard schedule in [45] for 90, 000 iterations with SGD [46] using the learning rate of 0.00125. We applied color [47], depth [48], and crop augmentation. Training took about 8h on a single Tesla A100, and the inference took 0.13 s per image (1,000 iterations) on a Titan XP.…”

Section: Hierarchical Occlusion Modelingmentioning

confidence: 99%

Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling

Back¹,

Lee²,

Kim³

et al. 2021

Preprint

View full text Add to dashboard Cite

Instance-aware segmentation of unseen objects is essential for a robotic system in an unstructured environment. Although previous works achieved encouraging results, they were limited to segmenting the only visible regions of unseen objects. For robotic manipulation in a cluttered scene, amodal perception is required to handle the occluded objects behind others. This paper addresses Unseen Object Amodal Instance Segmentation (UOAIS) to detect 1) visible masks, 2) amodal masks, and 3) occlusions on unseen object instances. For this, we propose a Hierarchical Occlusion Modeling (HOM) scheme designed to reason about the occlusion by assigning a hierarchy to a feature fusion and prediction order. We evaluated our method on three benchmarks (tabletop, indoors, and bin environments) and achieved state-of-the-art (SOTA) performance. Robot demos for picking up occluded objects, codes, and datasets are available at https://sites.google.com/view/uoais.

show abstract

“…The answer for this case is domain randomization. Domain randomization is a popular approach [49,21,55,36,47,40] that aims to randomize parts of the domain that we do not want our algorithm to be sensitive to. For example, [49] and [40] trained complex recognition methods by means of adding variability to the input render data, i.e., different illumination conditions, texture changes, scene decomposition, etc.…”

Section: Domain Randomizationmentioning

confidence: 99%

“…This sort of parameterization allows to learn features that are invariant to the particular properties of the domain. The authors of [55] used a sophisticated depth augmentation pipeline trying to cover possible artifacts of the common commodity depth sensors. It was then used to train a network removing these artifacts from the input and generating a clean, synthetically-looking image.…”

Section: Domain Randomizationmentioning

confidence: 99%

DeceptionNet: Network-Driven Domain Randomization

Zakharov

Kehl²,

Ilić

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

We present a novel approach to tackle domain adaptation between synthetic and real data. Instead, of employing "blind" domain randomization, i.e., augmenting synthetic renderings with random backgrounds or changing illumination and colorization, we leverage the task network as its own adversarial guide toward useful augmentations that maximize the uncertainty of the output. To this end, we design a min-max optimization scheme where a given task competes against a special deception network to minimize the task error subject to the specific constraints enforced by the deceiver. The deception network samples from a family of differentiable pixel-level perturbations and exploits the task architecture to find the most destructive augmentations. Unlike GAN-based approaches that require unlabeled data from the target domain, our method achieves robust mappings that scale well to multiple target distributions from source data alone. We apply our framework to the tasks of digit recognition on enhanced MNIST variants, classification and object pose estimation on the Cropped LineMOD dataset as well as semantic segmentation on the Cityscapes dataset and compare it to a number of domain adaptation approaches, thereby demonstrating similar results with superior generalization capabilities.

show abstract

Keep it Unreal: Bridging the Realism Gap for 2.5D Recognition with Geometry Priors Only

Cited by 29 publications

References 26 publications

Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling

DeceptionNet: Network-Driven Domain Randomization

Contact Info

Product

Resources

About