Temporal Segment Networks for Action Recognition in Videos

Abstract

Deep convolutional networks have achieved great success for image recognition. However, for action recognition in videos, their advantage over traditional methods is not so evident. We present a general and flexible video-level framework for learning action models in videos. This method, called temporal segment network (TSN), aims to model long-range temporal structures with a new segment-based sampling and aggregation module. This unique design enables our TSN to efficiently learn action models by using the whole action videos. The learned models could be easily adapted for action recognition in both trimmed and untrimmed videos with simple average pooling and multi-scale temporal window integration, respectively. We also study a series of good practices for the instantiation of TSN framework given limited training samples. Our approach obtains the state-the-of-art performance on four challenging action recognition benchmarks: HMDB51 (71.0%), UCF101 (94.9%), THUMOS14 (80.1%), and ActivityNet v1.2 (89.6%). Using the proposed RGB difference for motion models, our method can still achieve competitive accuracy on UCF101 (91.0 %) while running at 340 FPS. Furthermore, based on the temporal segment networks, we won the video classification track at the ActivityNet challenge 2016 among 24 teams, which demonstrates the effectiveness of TSN and the proposed good practices.

Learning Spatio-Temporal Representation With Local and Global Diffusion

“…In [34], the famous two-stream architecture is devised by applying two 2D CNN architectures separately on visual frames and staked optical flows. This two-stream architecture is further extended by exploiting convolutional fusion [5], spatio-temporal attention [24], temporal segment networks [41,42] and convolutional encoding [4,27] for video representation learning. Ng et al [49] highlight the drawback of performing 2D CNN on video frames, in which long-term dependencies cannot be captured by two-stream network.…”

Explainable Video Action Reasoning via Prior Knowledge and State Transitions

“…Video action recognition. Without rules for logical reasoning, many approaches often employ hand-crafted [19,24,34,43] or deeplearned features [8,9,23,36,44,45] of appearance and motion for action recognition. Recently, researchers attempt to use the semantic-level state changes [1,7,10,25,49,50] for video analysis.…”

“…Note that there are essential differences between the proposed action reasoning approach and many deep learning based action recognition methods [8,9,23,36,44,45]: (1) Instead of only predicting a single action label, our method outputs multiple action labels with relevant objects, attributes/relationships and the time of each state transition. (2) Our action models are learned from semanticlevel state transitions based definitions (state detectors are trained on still images), and thus it does not need well-annotated video clips for training.…”

“…We compare our method to a representative two-stream (appearance and motion) action recognition algorithm TSN [45], which adopts an end-to-end deep learning scheme that utilizes RGB frames and optical flow as a two-stream input. It is worth mentioning that TSN achieves the state-of-the-art performance 94.9% and 89.6% on the benchmark action recognition dataset UCF 101 [38] and ActivityNet [6], respectively.…”

Temporal Accumulative Features for Sign Language Recognition

“…Recently, I3D architecture [9] was proposed as an improvement of two-stream networks. In another work, Wang et al [53] have proposed Temporal Segment Networks (TSNs) with the purpose of solving the long-range temporal limitations of twostream networks by using temporal sampling. More recently, Choutas et al [12] proposed PoTion representations for human action recognition.…”