High-Level Fusion of Depth and Intensity for Pedestrian Classification

Abstract

Abstract. This paper presents a novel approach to pedestrian classification which involves a high-level fusion of depth and intensity cues. Instead of utilizing depth information only in a pre-processing step, we propose to extract discriminative spatial features (gradient orientation histograms and local receptive fields) directly from (dense) depth and intensity images. Both modalities are represented in terms of individual feature spaces, in each of which a discriminative model is learned to distinguish between pedestrians and non-pedestrians. We refrain from the construction of a joint feature space, but instead employ a high-level fusion of depth and intensity at classifier-level. Our experiments on a large real-world dataset demonstrate a significant performance improvement of the combined intensity-depth representation over depth-only and intensity-only models (factor four reduction in false positives at comparable detection rates). Moreover, high-level fusion outperforms low-level fusion using a joint feature space approach.

The Benefits of Dense Stereo for Pedestrian Detection

“…With regard to pedestrian classification, we extract spatial features from dense depth images at medium resolution (pedestrian heights up to 80 pixels) and fuse them with an intensity-based feature set on the classifier level. This paper builds upon our earlier work [23], [37] and presents an integrated pedestrian system that significantly outperforms the state of the art.…”

“…or/and modalities (intensity, depth, motion, etc.) into a single pattern classification module [8], [33], [37], [38], [43], [46], [48]. One fusion approach involves integration of all cues into a single joint feature space [38], [43], [46].…”

“…Boosting approaches have also been proposed to automatically select the "best" features from a pool of different features and modalities [46], [48]. In contrast, [8], [33], and [37] utilize fusion on the classifier level by training a specialized classifier for each feature or modality. Classifier fusion is done using fuzzy integration [33], simple classifier combination rules [37], or a mixture-of-experts framework [8].…”

“…We follow a classifier-level fusion strategy that bases the final decision on a combined vote of the individual classifiers. As opposed to fusion approaches using a joint feature space, e.g., [38], [43], and [46], this strategy does not suffer from the increased dimensionality of the joint space; see [37] and [38]. We assume our approach to generalize to other state-ofthe-art features and classifiers, which are complex enough to capture the appearance of the pedestrian class; see [9].…”

“…Each feature set is then used as input to a single classifier, and their outputs are combined. As opposed to creating a joint feature space, classifier-level fusion does not suffer from effects related to the increased dimensionality of the joint space; see [37] and [38].…”