An information theoretic approach for tracker performance evaluation

Abstract

Automated tracking of vehicles and people is essential for the effective utilization of imagery in wide area surveillance applications. In order to determine the best tracking algorithm and parameters for a given application, a comprehensive evaluation procedure is required. However, despite half a century of research in multi-target tracking, there is no consensus on how to score the overall performance of these trackers. Existing evaluation approaches assess tracker performance through measures of correspondence between ground truth tracks and system tracks using metrics such as track detection rate, track completeness, track fragmentation rate, and track ID change rate. However, each of these only provides a partial measure of performance and no good method exists to combine them into a holistic metric. Towards this end, this paper presents a pair of information theoretic metrics with similar behavior to the Receiver Operating Characteristic (ROC) curves of signal detection theory. Overall performance is evaluated with the percentage of truth information that a tracker captured and the total amount of false information that it reported. Information content is quantified through conditional entropy and mutual information computations using numerical estimates of the probability of association between the truth and the system tracks. This paper demonstrates how these information quality metrics provide a comprehensive evaluation of overall tracker performance and how they can be used to perform tracker comparisons and parameter tuning on wide-area surveillance imagery and other applications. 1

Is my new tracker really better than yours?

“…Until recently the majority of papers that address performance evaluation in visual tracking were concerned with multi-target tracking scenarios [32,17,8,9,10,25,6,24]. One might view the multi-target tracking as a generalization of single-target tracking, however, there is a crucial difference in the focus of the evaluation.…”

Local Metrics for Multi-Object Tracking

“…While the application governs which type of metric is suitable, each has associated drawbacks. For non-strict metrics, it is unclear how best to define a scalar measure of association quality, leading to a proliferation of non-strict metrics [60,4,27,36,43,18]. Moreover, when combined with a detection metric, each association metric induces an implicit trade-off between detection and association which may not be well understood.…”

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

“…A number of other metrics (Shitrit et al 2011;Bento and Zhu 2016;Rahmathullah et al 2016;Edward et al 2009;Smith et al 2005;Wu et al 2017) have been been proposed for MOT evaluation, but to the best of our knowledge none of them have been adopted by any MOT benchmarks and thus have not become widely used for evaluation.…”