Adaptive visual tracking using the prioritized Q-learning algorithm: MDP-based parameter learning approach

Abstract

a b s t r a c tThis paper introduces an adaptive visual tracking method that combines the adaptive appearance model and the optimization capability of the Markov decision process. Most tracking algorithms are limited due to variations in object appearance from changes in illumination, viewing angle, object scale, and object shape. This paper is motivated by the fact that tracking performance degradation is caused not only by changes in object appearance but also by the inflexible controls of tracker parameters. To the best of our knowledge, optimization of tracker parameters has not been thoroughly investigated, even though it critically influences tracking performance. The challenge is to equip an adaptive tracking algorithm with an optimization capability for a more flexible and robust appearance model. In this paper, the Markov decision process, which has been applied successfully in many dynamic systems, is employed to optimize an adaptive appearance model-based tracking algorithm. The adaptive visual tracking is formulated as a Markov decision process based dynamic parameter optimization problem with uncertain and incomplete information. The high computation requirements of the Markov decision process formulation are solved by the proposed prioritized Q-learning approach. We carried out extensive experiments using realistic video sets, and achieved very encouraging and competitive results.

Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learning

“…The resulting partially-observable MDP (POMDP) violates Markov independence assumptions: actions depend on the entire history of observations rather than just the current one [20,47]. As in [16,23], we account for this partial observability by maintaining a memory that captures beliefs about the world, which we update over time (Sec. 3).…”

“…Interestingly, such an "end-to-end" learning philosophy is often embraced by the multi-object tracking community, where strategies for online reinitialization and data association are learned from data [26,62,31]. Most related to us are [62], who use an MDP for multi-object tracking, and [23], who use RL for single target tracking. Both works use heuristics to reduce policy learning to a supervised learning task, avoiding the need to reason about rewards in the far future.…”