Fast Learning of Temporal Action Proposal via Dense Boundary Generator

Abstract

Generating temporal action proposals remains a very challenging problem, where the main issue lies in predicting precise temporal proposal boundaries and reliable action confidence in long and untrimmed real-world videos. In this paper, we propose an efficient and unified framework to generate temporal action proposals named Dense Boundary Generator (DBG), which draws inspiration from boundary-sensitive methods and implements boundary classification and action completeness regression for densely distributed proposals. In particular, the DBG consists of two modules: Temporal boundary classification (TBC) and Action-aware completeness regression (ACR). The TBC aims to provide two temporal boundary confidence maps by low-level two-stream features, while the ACR is designed to generate an action completeness score map by high-level action-aware features. Moreover, we introduce a dual stream BaseNet (DSB) to encode RGB and optical flow information, which helps to capture discriminative boundary and actionness features. Extensive experiments on popular benchmarks ActivityNet-1.3 and THUMOS14 demonstrate the superiority of DBG over the state-of-the-art proposal generator (e.g., MGG and BMN).

Multi-scale Dynamic Network for Temporal Action Detection

“…Edge Suppression Loss. To classify each candidate video segment based on whether it contains complete action instance, previous methods [21,23,37] treat the video segments with a large Intersection over Union (IoU) as positive examples and the video segments with a small IoU as negative examples. In fact, for some negative examples, they also contain some action information and are difficult to be classified accurately among these methods.…”

“…To demonstrate the effectiveness of the proposed MDN, we compare it with more than 10 state-of-the-art temporal action detection algorithms, inluding DBG (AAAI'20) [21], G-TAD (CVPR'20) [37], BC-GNN (ECCV'20) [1], BU-TAL (ECCV'20) [39], TSI (ACCV'20) [25], etc. For a fair comparison, we use the same video feature representation and post-processing step.…”

“…It has attracted extensive attention recently and plays an important role in video captioning, video moment retrieval, visual question answering, and many other applications. Lately, a group of approaches based on Boundary-Matching (BM) mechanism [21,23,37] first extract the features of all candidate video segments in the untrimmed video to get the segment-level feature map, in which each element potentially contains a background segment or a to-be-detected action instance, and then classify these video segments to get the final detection results. Though these approaches [21,23,37] have made remarkable progress in TAD with their ability to extract richer contextual semantic features, they still suffer from some drawbacks.…”

“…Lately, a group of approaches based on Boundary-Matching (BM) mechanism [21,23,37] first extract the features of all candidate video segments in the untrimmed video to get the segment-level feature map, in which each element potentially contains a background segment or a to-be-detected action instance, and then classify these video segments to get the final detection results. Though these approaches [21,23,37] have made remarkable progress in TAD with their ability to extract richer contextual semantic features, they still suffer from some drawbacks. Firstly, different input videos often contain different action and background information, and using the same convolution kernels to process them is obviously not reasonable.…”

Reasoning over multiple human-human interaction activities

“…However, through a simple sliding-window strategy, our method can be applied without modification to classify actions with unknown length in untrimmed videos. Nonetheless, using segmented videos would lead to a better performance, and these can be obtained by employing action proposal methods [101][102][103].…”