Simulations for Validation of Vision Systems

Abstract

As the computer vision matures into a systems science and engineering discipline, there is a trend in leveraging latest advances in computer graphics simulations for performance evaluation, learning, and inference. However, there is an open question on the utility of graphics simulations for vision with apparently contradicting views in the literature. In this paper, we place the results from the recent literature in the context of performance characterization methodology outlined in the 90's and note that insights derived from simulations can be qualitative or quantitative depending on the degree of fidelity of models used in simulation and the nature of the question posed by the experimenter. We describe a simulation platform that incorporates latest graphics advances and use it for systematic performance characterization and tradeoff analysis for vision system design. We verify the utility of the platform in a case study of validating a generative model inspired vision hypothesis, Rank-Order consistency model, in the contexts of global and local illumination changes, and bad weather, and high-frequency noise. Our approach establishes the link between alternative viewpoints, involving models with physics based semantics and signal and perturbation semantics and confirms insights in literature on robust change detection.

Advancing Non-Contact Vital Sign Measurement using Synthetic Avatars

“…Using parameterized graphics simulations to augment existing datasets have been extensively explored in different computer vision domains [43,48,49,50,47,28,17] such as training pose recognition [43], scene segmentation for selfdriving cars [38], improving object recognition [34], detect-ing pedestrians under different conditions [47], and for performance evaluation of learned models [17]. AirSim is a graphics-based simulation environment [40] that has been successfully used in the context of training autonomous drone navigation [6] and the systematic evaluation of face detection systems [28].…”

Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing

“…There is a long history of the use of synthetic data in training and evaluating computer vision systems [23], [24], [25], [26], [27], [28], [29], [30], [7], [5], [8], [4], [6]. Synthetics have been employed extensively in models for face and body analysis specifically [31], [7], [28], [29], [32].…”

Perception robustness testing at different levels of generality

“…Another popular tool for selecting algorithms is using simulation, with many available datasets, such as the public Virtual KITTI 2.0 (Gaidon et al, 2016) and SYNTHIA (Johnson-Roberson et al, 2017), to provide testing environments. While it has been shown that performance of models in simulation can be quite different than in the real world, the performance trends and failure modes of models still hold (Veeravasarapu et al, 2015). Higher fidelity physical modeling can be a solution here and is described by the various major, commercial, autonomous car developers (Argo AI, 2019;Baidu AI, 2017;Bigelow, 2019;Madrigal, 2017;Nvidia, 2018;Reynolds, 2019), but can require significant resources and effort to capture the variation of the real world adequately, which is important for algorithms that are looking for complex patterns.…”