Large-Scale Object Detection of Images from Network Cameras in Variable Ambient Lighting Conditions

Abstract

Computer vision relies on labeled datasets for training and evaluation in detecting and recognizing objects. The popular computer vision program, YOLO ("You Only Look Once"), has been shown to accurately detect objects in many major image datasets. However, the images found in those datasets, are independent of one another and cannot be used to test YOLO's consistency at detecting the same object as its environment (e.g. ambient lighting) changes. This paper describes a novel effort to evaluate YOLO's consistency for large-scale applications. It does so by working (a) at large scale and (b) by using consecutive images from a curated network of public video cameras deployed in a variety of real-world situations, including traffic intersections, national parks, shopping malls, university campuses, etc. We specifically examine YOLO's ability to detect objects in different scenarios (e.g., daytime vs. night), leveraging the cameras' ability to rapidly retrieve many successive images for evaluating detection consistency. Using our camera network and advanced computing resources (supercomputers), we analyzed more than 5 million images captured by 140 network cameras in 24 hours. Compared with labels marked by humans (considered as "ground truth"), YOLO struggles to consistently detect the same humans and cars as their positions change from one frame to the next; it also struggles to detect objects at night time. Our findings suggest that stateof-the art vision solutions should be trained by data from network camera with contextual information before they can be deployed in applications that demand high consistency on object detection.

Why Accuracy Is Not Enough: The Need for Consistency in Object Detection

“…Two images of the same scene can be captured by the same camera less than a second apart, yet the predictions of a neural network on those two visually similar images can differ dramatically. The small differences between the two images are called image distortions, and they can be caused by a range of factors, including ambient light level and camera sensor noise [13].…”

“…As Tung, et al [13] observe, popular image datasets (e.g., ImageNet, Microsoft COCO) are filled with visually dissimilar images. Thus, those datasets are largely unsuitable for consistency testing.…”

“…If IoU = 0, the boxes have no overlap. In literature, an IoU threshold of 0.5 is typically used [13] to decide if two bounding boxes sufficiently overlap.…”

“…4) Unsharp Mask (UM). Tung, et al [13] explain that if an object is moving, the motion blur can make it harder for a network to extract…”

Robot perceptual adaptation to environment changes for long-term human teammate following

“…However, these methods are specifically implemented for place recognition or robot localization problems, and cannot be applied to appropriate perception of objects of interest under short- and long-term context or environment changes. Tung et al (2019) evaluated YOLO’s (Redmon et al, 2016) performance during long-term changes on detecting an object of interest over long periods of time under various lighting conditions. It was observed that YOLO continuously struggles to detect the same object during sudden changes and during night time.…”