Human Synthesis and Scene Compositing

Zanfir, Mihai; Oneata, Elisabeta; Popa, Alin-Ionut; Zanfir, Andrei; Sminchisescu, Cristian

doi:10.1609/aaai.v34i07.6969

Cited by 22 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…A recent promising direction synthesizes realistic looking people in images [57,61]. Zanfir et al [57] use a learned human synthesis method to insert generated people in images such that they make sense relative to the scene geometry and lighting. While they can condition the generated person on pose and shape, the resulting images contain artifacts that are common to generative models, making the results unsuitable as ground truth.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

AGORA: Avatars in Geography Optimized for Regression Analysis

Patel¹,

Huang²,

Tesch³

et al. 2021

Preprint

View full text Add to dashboard Cite

includes 257 scans of children. We create reference 3D poses and body shapes by fitting the SMPL-X body model (with face and hands) to the 3D scans, taking into account clothing. We create around 14K training and 3K test images by rendering between 5 and 15 people per image using either image-based lighting or rendered 3D environments, taking care to make the images physically plausible and photoreal. In total, AGORA consists of 173K individual person crops. We evaluate existing state-of-theart methods for 3D human pose estimation on this dataset. and find that most methods perform poorly on images of children. Hence, we extend the SMPL-X model to better capture the shape of children. Additionally, we finetune methods on AGORA and show improved performance on both AGORA and 3DPW, confirming the realism of the dataset. We provide all the registered 3D reference training data, rendered images, and a web-based evaluation site at https://agora.is.tue.mpg.de/.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

AGORA: Avatars in Geography Optimized for Regression Analysis

Patel¹,

Huang²,

Tesch³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Others have also considered detailed 3D modeling of people and the input scene (Zanfir et al 2020), or modeling individual component attributes (e.g., mixing clothing items such as shorts and a tank top from different input photographs (Men et al 2020). While the outputs of these models are certainly impressive, their focus (and their evaluation) is on generating visually pleasing fashion photographs in, typically, indoor scenes with a limited variation of imaging conditions.…”

Section: Related Work and Positioningmentioning

confidence: 99%

Artificial Dummies for Urban Dataset Augmentation

Vobecký

Hurych

Uřičář³

et al. 2021

AAAI

View full text Add to dashboard Cite

Existing datasets for training pedestrian detectors in images suffer from limited appearance and pose variation. The most challenging scenarios are rarely included because they are too difficult to capture due to safety reasons, or they are very unlikely to happen. The strict safety requirements in assisted and autonomous driving applications call for an extra high detection accuracy also in these rare situations. Having the ability to generate people images in arbitrary poses, with arbitrary appearances and embedded in different background scenes with varying illumination and weather conditions, is a crucial component for the development and testing of such applications. The contributions of this paper are three-fold. First, we describe an augmentation method for the controlled synthesis of urban scenes containing people, thus producing rare or never-seen situations. This is achieved with a data generator (called DummyNet) with disentangled control of the pose, the appearance, and the target background scene. Second, the proposed generator relies on novel network architecture and associated loss that takes into account the segmentation of the foreground person and its composition into the background scene. Finally, we demonstrate that the data generated by our DummyNet improve the performance of several existing person detectors across various datasets as well as in challenging situations, such as night-time conditions, where only a limited amount of training data is available. In the setup with only day-time data available, we improve the night-time detector by 17% log-average miss rate over the detector trained with the day-time data only.

show abstract

“…Video generation is a challenging problem. GAN [34] based generative models are proposed benefiting from the development of computing power and achieve great success in image generation [35]- [41], but video generation is much harder for GANs due to more data contents and extra temporal information. Many works [14], [16], [18], [42] directly implement 3D convolutions to encode temporal information.…”

Section: Related Workmentioning

confidence: 99%

Pose-Forecasting Aided Human Video Prediction With Graph Convolutional Networks

Zhao

Dou

2020

IEEE Access

View full text Add to dashboard Cite

Human video prediction is still a challenging problem due to the uncertainty of future actions and complexity of frame details. Recent methods tackle this problem in two steps: firstly to forecast future human poses from the initial ones, and then to generate realistic frames conditioned on predicted poses. Following this framework, we propose a novel Graph Convolutional Network (GCN) based pose predictor to comprehensively model human body joints and forcast their positions holistically, and also a stacked generative model with a temporal discriminator to iteratively refine the quality of the generated videos. The GCN based pose predictor fully considers the relationships among body joints and produces more plausible pose predictions. With the guidance of predicted poses, a temporal discriminator encodes temporal information into future frame generation to achieve high-quality results. Furthermore, stacked residual refinement generators make the results more realistic. Extensive experiments on benchmark datasets demonstrate that the proposed method produces better predictions than state-of-the-arts and achieves up to 15% improvement in PSNR. INDEX TERMS Human video prediction, GCN, GAN, temporal discriminator, stacked residual refinment.

show abstract

Human Synthesis and Scene Compositing

Cited by 22 publications

References 28 publications

AGORA: Avatars in Geography Optimized for Regression Analysis

AGORA: Avatars in Geography Optimized for Regression Analysis

Artificial Dummies for Urban Dataset Augmentation

Pose-Forecasting Aided Human Video Prediction With Graph Convolutional Networks

Contact Info

Product

Resources

About