Registering Multimodal Imagery with Occluding Objects Using Mutual Information: Application to Stereo Tracking of Humans

Abstract

This chapter introduces and analyzes a method for registering multimodal images with occluding objects in the scene. An analysis of multimodal image registration gives insight into the limitations of assumptions made in current approaches and motivates the methodology of the developed algorithm. Using calibrated stereo imagery, we use maximization of mutual information in sliding correspondence windows that inform a disparity voting algorithm to demonstrate successful registration of objects in color and thermal imagery where there is significant occlusion. Extensive testing of scenes with multiple objects at different depths and levels of occlusion shows high rates of successful registration. Ground truth experiments demonstrate the utility of disparity voting techniques for multimodal registration by yielding qualitative and quantitative results that outperform approaches that do not consider occlusions. A framework for tracking with the registered multimodal features is also presented and experimentally validated. IntroductionComputer vision applications are increasingly using multimodal imagery to obtain and process information about a scene. Specifically, the disparate yet complementary nature of visual and thermal imagery has been used in recent works to obtain additional information and robustness [1,2]. The use of both types of imagery yields information about the scene that is rich in color, depth, motion, and thermal detail. Such information can then be used to successfully detect, track, and analyze people and objects in the scene.To associate the information from each modality, corresponding data in each image must be successfully registered. In long-range surveillance applications [2], the cameras are assumed to be oriented in such a way that a global alignment R.I. Hammoud (ed.), Augmented Vision Perception in Infrared: Algorithms and 321

Improved Depth Recovery In Consumer Depth Cameras via Disparity Space Fusion within Cross-spectral Stereo

“…pedestrians) via prior foreground extraction and subsequent localised stereo matching [12,13]. Krotosky and Trivedi [13] additionally demonstrate the failure of dense depth computation using MI in the global energy minimisation framework of [8] caused by the lack of a global intensity transform between the images. Torabi and Bilodeau [20] describe a very similar windowbased approach but replace MI by Local Self-Similarity (LSS) as a correspondence measure.…”

“…Work on the conventional cross-spectral stereo problem, outside of the specifics of RGB-D depth recovery improvement, is more developed [12,13,15,20]. Krotosky and Trivedi [12,13] investigate cross-spectral stereo for pedestrian detection and tracking, using a windowbased Mutual Information (MI) approach inspired by the original work of Egnal [5].…”

“…Krotosky and Trivedi [12,13] investigate cross-spectral stereo for pedestrian detection and tracking, using a windowbased Mutual Information (MI) approach inspired by the original work of Egnal [5]. However, depth computation is only performed for isolated objects (i.e.…”

“…However, depth computation is only performed for isolated objects (i.e. pedestrians) via prior foreground extraction and subsequent localised stereo matching [12,13]. Krotosky and Trivedi [13] additionally demonstrate the failure of dense depth computation using MI in the global energy minimisation framework of [8] caused by the lack of a global intensity transform between the images.…”

“…More recently, Pinggera et al [15] introduce a new dense depth recovery approach using Histogram of Oriented Gradient (HOG) feature descriptors [4] to address the problem of cross-spectral stereo matching between colour and far-infrared images. Essentially HOG descriptors are computed for each pixel to capture local geometry, invariant to spectral image characteristics, and are shown in [15] to notably outperform prior work [13,20] and conventional radiometric (illumination) invariant stereo matching techniques [10].…”