Online Detection of Action Start in Untrimmed, Streaming Videos

Abstract

We aim to tackle a novel task in action detection -Online Detection of Action Start (ODAS) in untrimmed, streaming videos. The goal of ODAS is to detect the start of an action instance, with high categorization accuracy and low detection latency. ODAS is important in many applications such as early alert generation to allow timely security or emergency response. We propose three novel methods to specifically address the challenges in training ODAS models: (1) hard negative samples generation based on Generative Adversarial Network (GAN) to distinguish ambiguous background, (2) explicitly modeling the temporal consistency between data around action start and data succeeding action start, and (3) adaptive sampling strategy to handle the scarcity of training data. We conduct extensive experiments using THUMOS'14 and ActivityNet. We show that our proposed methods lead to significant performance gains and improve the state-of-the-art methods. An ablation study confirms the effectiveness of each proposed method.

An empirical study on temporal modeling for online action detection

“…Eun et al [15] designed a novel recurrent unit named Information Discrimination Unit to explicitly discriminate the information relevant to an ongoing action from others. Besides, Shou et al [56] formulated the online detection of action start (ODAS) as a classification task of sliding windows and introduced a model based on Generative Adversarial Network (GAN) to generate hard negative samples to improve the training of the samples. Gao et al [21] proposed StartNet to address ODAS which can be decomposed into two stages: action classification and start point localization.…”

Understanding Videos at Scale: How to Extract Insights for Business Research

“…Initial evidence suggests that the types of scene cuts in a video can have important effects on video success (Liu et al, 2018). Scene cut detection can be accomplished with the open-source solution PySceneDetect (Almousa et al, 2018;Shou et al, 2018), which is integrated into our tool. The algorithm fundamentally compares the hue, saturation, and value of two neighboring frames and compares the average distances against a threshold.…”

DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization

“…Since the number of video frames is large (e.g., each video in TACoS has average 9,000 frames), but the positive training samples are sparse, i.e., only two frames (Figure 2 (b)). 3) Detecting temporal action boundary from frames, i.e., predicting a frame is queryrelated and at temporal boundary simultaneously by a single network, is still a challenging task, even without query constraint (Shou et al, 2018).…”