Self-Supervised Object Detection via Generative Image Synthesis

Mustikovela, Siva Karthik; Mello, Shalini De; Prakash, Aayush; Iqbal, Umar; Liu, Sifei; Nguyen-Phuoc, Thu; Rother, Carsten; Kautz, Jan

doi:10.1109/iccv48922.2021.00849

Cited by 10 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternative methods use point-based differentiable rendering [54,3] and can optimize over scene geometry, camera model, and various image formation properties. While these methods overfit to specific scenes, recent self-supervised approaches learn generative models of specific objects [41] and can render novel and controllable complex scenes by exploiting compositionality [43]. While neural volume rendering and point based techniques can yield impressive results, other methods aim to explicitly model various parts of traditional graphics pipelines [53,42,2,28,58].…”

Section: Related Workmentioning

confidence: 99%

“…In the context of object detection, CAD models are typically assumed known [22,68,49] and a subset of lighting, textures, materials, and object poses are randomized. Although typically inefficient, sample efficiency can be improved via differentiable guided augmentations [67], while content [29,10] and appearance [51,41] gaps can also be addressed by leveraging real data. However, a significant gap remains in terms of the photo-realism of the images generated.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Photo-realistic Neural Domain Randomization

Zakharov¹,

Ambruş²,

Guizilini³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Synthetic data is a scalable alternative to manual supervision, but it requires overcoming the sim-to-real domain gap. This discrepancy between virtual and real worlds is addressed by two seemingly opposed approaches: improving the realism of simulation or foregoing realism entirely via domain randomization. In this paper, we show that the recent progress in neural rendering enables a new unified approach we call Photo-realistic Neural Domain Randomization (PNDR). We propose to learn a composition of neural networks that acts as a physics-based ray tracer generating high-quality renderings from scene geometry alone. Our approach is modular, composed of different neural networks for materials, lighting, and rendering, thus enabling randomization of different key image generation components in a differentiable pipeline. Once trained, our method can be combined with other methods and used to generate photo-realistic image augmentations online and significantly more efficiently than via traditional ray-tracing. We demonstrate the usefulness of PNDR through two downstream tasks: 6D object detection and monocular depth estimation. Our experiments show that training with PNDR enables generalization to novel scenes and significantly outperforms the state of the art in terms of real-world transfer.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Photo-realistic Neural Domain Randomization

Zakharov¹,

Ambruş²,

Guizilini³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Nowadays, models (pre)trained on synthetic datasets have a broad range of utility including feature matching (DeTone et al, 2018) autonomous driving (Siam et al, 2021), robotics indoor and aerial navigation , scene segmentation (Roberts et al, 2021) and anonymized image generation in healthcare (Piacentino et al, 2021). The approaches broadly adopt the following process: pre-train with synthetic data before training on real-world scenes (DeTone et al, 2018;Hinterstoisser et al, 2019), generate composites of synthetic data and real images to create a new one that contains the desired representation (Hinterstoisser et al, 2018) or generate realistic datasets using simulation engines like Unity (Borkman et al, 2021) or generative models like GANs (Jeon et al, 2021;Mustikovela et al, 2021). There are limitations to each of these regimes but one of the most common pitfalls is performance deterioration in real-world datasets.…”

Section: Synthetic Data Generationmentioning

confidence: 99%

Knock, knock. Who's there? -- Identifying football player jersey numbers with synthetic data

Bhargavi¹,

Coyotl²,

Gholami³

2022

Preprint

View full text Add to dashboard Cite

Automatic player identification is an essential and complex task in sports video analysis. Different strategies have been devised over the years, but identification based on jersey numbers is one of the most common approaches given its versatility and relative simplicity. However, automatic detection of jersey numbers is still challenging due to changing camera angles, low video resolution, small object size in wide-range shots and transient changes in the player's posture and movement. In this paper we present a novel approach for jersey number identification in a small, highly imbalanced dataset from the Seattle Seahawks practice videos. We use a multi-step strategy that enforces attention to a particular region of interest (player's torso), to identify jersey numbers. We generate in-house synthetic datasets of different complexities to supplement the data imbalance and scarcity in the samples. Our multi-step pipeline first identifies and crops players in a frame using a pretrained person detection model. We then utilize a pretrained human pose estimation model to localize jersey numbers (using torso key-points) in the detected players, obviating the need for annotating bounding boxes for number detection. This results in images that are on average 20x25px in size. We trained two light-weight Convolutional Neural Networks (CNNs) with different learning objectives: multi-class for two-digit number identification and multi-label for digit-wise detection to compare performance. Both models went through a pre-training round with the synthetic datasets and were finetuned with the real-world dataset to achieve a final best accuracy of 89%. Our results indicate that simple models can achieve an acceptable performance on the jersey number detection task and that synthetic data can improve the performance dramatically (accuracy increase of 9% overall, 18% on low frequency numbers) making our approach achieve state of the art results.

show abstract

“…Nowadays, models (pre)-trained on synthetic datasets have a broad range of utility including feature matching (DeTone et al, 2018 ) autonomous driving (Siam et al, 2021 ), robotics indoor and aerial navigation (Nikolenko, 2021a ), scene segmentation (Roberts et al, 2021 ), and anonymized image generation in healthcare (Piacentino et al, 2021 ). The approaches broadly adopt the following process: pre-train with synthetic data before training on real-world scenes (DeTone et al, 2018 ; Hinterstoisser et al, 2019 ), generate composites of synthetic data and real images to create a new one that contains the desired representation (Hinterstoisser et al, 2018 ) or generate realistic datasets using simulation engines like Unity (Borkman et al, 2021 ) or generative models like GANs (Jeon et al, 2021 ; Mustikovela et al, 2021 ). There are limitations to each of these regimes but one of the most common pitfalls is performance deterioration in real-world datasets.…”

Section: Related Workmentioning

confidence: 99%

Jersey number detection using synthetic data in a low-data regime

Bhargavi

Gholami²,

Coyotl³

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Player identification is an essential and complex task in sports video analysis. Different strategies have been devised over the years and identification based on jersey numbers is one of the most common approaches given its versatility and relative simplicity. However, automatic detection of jersey numbers is challenging due to changing camera angles, low video resolution, small object size in wide-range shots, and transient changes in the player's posture and movement. In this paper, we present a novel approach for jersey number identification in a small, highly imbalanced dataset from the Seattle Seahawks practice videos. We generate novel synthetic datasets of different complexities to mitigate the data imbalance and scarcity in the samples. To show the effectiveness of our synthetic data generation, we use a multi-step strategy that enforces attention to a particular region of interest (player's torso), to identify jersey numbers. The solution first identifies and crops players in a frame using a person detection model, then utilizes a human pose estimation model to localize jersey numbers in the detected players, obviating the need for annotating bounding boxes for number detection. We experimented with two sets of Convolutional Neural Networks (CNNs) with different learning objectives: multi-class for two-digit number identification and multi-label for digit-wise detection to compare performance. Our experiments indicate that our novel synthetic data generation method improves the accuracy of various CNN models by 9% overall, and 18% on low frequency numbers.

show abstract

Self-Supervised Object Detection via Generative Image Synthesis

Cited by 10 publications

References 27 publications

Photo-realistic Neural Domain Randomization

Photo-realistic Neural Domain Randomization

Knock, knock. Who's there? -- Identifying football player jersey numbers with synthetic data

Jersey number detection using synthetic data in a low-data regime

Contact Info

Product

Resources

About