A Systematic Evaluation and Benchmark for Person Re-Identification: Features, Metrics, and Datasets

Abstract

Person re-identification (re-id) is a critical problem in video analytics applications such as security and surveillance. The public release of several datasets and code for vision algorithms has facilitated rapid progress in this area over the last few years. However, directly comparing re-id algorithms reported in the literature has become difficult since a wide variety of features, experimental protocols, and evaluation metrics are employed. In order to address this need, we present an extensive review and performance evaluation of single- and multi-shot re-id algorithms. The experimental protocol incorporates the most recent advances in both feature extraction and metric learning. To ensure a fair comparison, all of the approaches were implemented using a unified code library that includes 11 feature extraction algorithms and 22 metric learning and ranking techniques. All approaches were evaluated using a new large-scale dataset that closely mimics a real-world problem setting, in addition to 16 other publicly available datasets: VIPeR, GRID, CAVIAR, DukeMTMC4ReID, 3DPeS, PRID, V47, WARD, SAIVT-SoftBio, CUHK01, CHUK02, CUHK03, RAiD, iLIDSVID, HDA+ and Market1501. The evaluation codebase and results will be made publicly available for community use.

The empirical moment matrix and its application in computer vision

“…In four datasets, we achieve new state-of-the-art performance on CUHK01 and in bounding boxes that may not accurately represent a human [91]. As noted in our recent benchmark paper [92], the size of a dataset, in terms of both number of identities as well as number of bounding boxes, is critical to achieve good performance. Furthermore, in real-world end-to-end surveillance systems, as noted in Camps et al [91], we can use camera calibration information to predict motion patterns, potentially helping to prune out irrelevant candidates and reducing the search space.…”

“…2,798 false positives and 500k distractors, providing for a realistic gallery. Airport [92] represents a realistic scenario in which 1,382 identities are captured in a 6-camera indoor surveillance network in an airport. All images are automatically generated by means of an end-to-end re-id system [91,109].…”

“…Following the protocol described in [92], we evaluated 7 different feature extraction algorithms published up through CVPR 2016 (Table 5.3), which we briefly describe next. ELF Table 5.4 lists all the metric learning methods that were evaluated, which we briefly describe next.…”

“…Recently, instead of assuming perfect Bounding boxes (BBox) and trajectories from manual labeling, several end-to-end re-ID datasets with automatically person detector and tracking modules were released to test the robustness of new methods. For more details, we refer the reader to [68,69,70,71].…”

“…Recently, deep neural networks have been adopted to learn both the descriptor and classifier simultaneously [63,82,61,64]. For more details, we refer the reader to [68,69,70,71].…”