Simulated gold-standard for quantitative evaluation of monocular vision algorithms

Abstract

In the physical universe, truth for computer vision (CV) is impractical if not impossible to obtain. As a result, the CV community has resorted to qualitative practices and sub-optimal quantitative measures. This is problematic because it limits our ability to train, evaluate, and ultimately understand algorithms such as single image depth estimation (SIDE) and structure from motion (SfM). How good are these algorithms, individually and relatively, and where do they break? Herein, we discuss that while truth evades both the real and simulated (SIM) universes, a SIM CV gold-standard can be achieved. We outline an extensible SIM framework and data collection workflow using unreal engine with the Robot Operating System (ROS) for three-dimensional mapping on low altitude aerial vehicles. Furthermore, voxel-based mapping measures from algorithm output to a SIM gold-standard are discussed. The proposed metrics are demonstrated by analyzing performance across changes in platform context. Ultimately, the current article is a step towards an improved process for comparing algorithms, evaluating their strengths and weaknesses, and automating algorithm design.

MizSIM: a headless open-source simulation framework for training and evaluating artificial intelligence

“…Our research conducted at the University of Missouri extensively leverages the UE across diverse areas of study. Notably, UE has played a pivotal role in our exploration of explainable AI, [12][13][14][15] procedural simulation for AI, [16][17][18] workflows enhancing computer vision, 17,[19][20][21][22][23] and multi-criteria decision making. 24,25 Additionally, UE has been instrumental in specific applications such as explosive hazard detection 14,18,26,27 and passive ranging.…”

How should simulated data be collected for AI/ML and unmanned aerial vehicles?

“…Recently, we introduced a SIM framework and workflows to help understand and characterize hand crafted and data-driven ML SfM algorithms. 49 We used the various data layers, truth, and metadata from SIM for evaluation at each image and across images to build a gold standard and ultimately focused metrics that compare and help us select algorithm parameters and ideal flight contexts (altitude, camera pose, etc.). We also used this LCAP capability to capture uncertainty in monocular depth estimation by constructing fuzzy voxel maps.…”