Searching Action Proposals via Spatial Actionness Estimation and Temporal Path Inference and Tracking

Abstract

Abstract. In this paper, we address the problem of searching action proposals in unconstrained video clips. Our approach starts from actionness estimation on frame-level bounding boxes, and then aggregates the bounding boxes belonging to the same actor across frames via linking, associating, tracking to generate spatial-temporal continuous action paths. To achieve the target, a novel actionness estimation method is firstly proposed by utilizing both human appearance and motion cues. Then, the association of the action paths is formulated as a maximum set coverage problem with the results of actionness estimation as a priori. To further promote the performance, we design an improved optimization objective for the problem and provide a greedy search algorithm to solve it. Finally, a tracking-by-detection scheme is designed to further refine the searched action paths. Extensive experiments on two challenging datasets, UCF-Sports and UCF-101, show that the proposed approach advances state-of-the-art proposal generation performance in terms of both accuracy and proposal quantity.

YoTube: Searching Action Proposal Via Recurrent and Static Regression Networks

“…For UCF-101 dataset, our method out-performs the state-ofthe-art [15] by 20% or more in all range of IoU. Although Li et al use deep network (RPN) [19], they use only one stream and their performance is only 4% better than the unsupervised method APT [14] in terms of the recall as shown in Table III.…”

“…Although aforementioned methods have greatly advanced the quality of action proposal, they still have limitations. Specifically, most of these works [3]- [5], [12], [14] either produce action proposals frame-by-frame individually which ignores the interplay between appearance, motion and temporal context among adjacent frames or arrange the spatial information learning and temporal context learning into isolated processes [15], [29] which produce less satisfactory results. Moreover, most of these methods work on trimmed videos [3]- [5], [14].…”

“…One limitation of existing deep learning based action proposals methods is they either process a video frame-byframe [5] or separate spatial and temporal information learning into two isolated processes [15], [29]. On the other hand, temporal dynamics and contexts among adjacent frames have been proven to be useful for recent state-of-the-arts in action classification and video description [1].…”

“…: OV (d j , g i ) ≥ η where η ∈ [0, 1]. In our work, we target to maximize the recall at a 0.5 threshold as other works [14], [15].…”

“…Brox eccv-10 [27] Jain cvpr-14 [3] Oneata eccv-14 [4] Gkioxari cpr-15 [5] APT bmvc-15 [14] Li accv-16 [15] YoTube (FLOW + RGB)…”