Asynchronous Temporal Fields for Action Recognition

Sigurdsson, Gunnar A.; Divvala, Santosh; Farhadi, Ali; Gupta, Abhinav

doi:10.1109/cvpr.2017.599

Cited by 138 publications

(77 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…60 DAP [13] 134.1 Single-stream R-C3D (Titan X Maxwell) [32] 569 Single-stream R-C3D (Titan X Pascal) [32] 1030 Two-stream R-C3D (Concat) (Titan X Pascal) * 656 Two-stream R-C3D (Sum) (Titan X Pascal) * 642 Single-stream R-C3D (OHEM) (Titan X Pascal) 1030 aged across 20 frames leading to more spatial consistency. As shown in Table VIII, our jointly optimized single stream model outperforms the asynchronous temporal fields model [61] as well as several baselines reported in the same paper [61]. While the improvement over the standard method is as high as 2.8%, the improvement after the post-processing is not as high.…”

Section: Fps S-cnn [3]mentioning

confidence: 59%

See 1 more Smart Citation

Two-Stream Region Convolutional 3D Network for Temporal Activity Detection

Das

Saenko

2019

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We address the problem of temporal activity detection in continuous, untrimmed video streams. This is a difficult task that requires extracting meaningful spatio-temporal features to capture activities, accurately localizing the start and end times of each activity. We introduce a new model, Region Convolutional 3D Network (R-C3D), which encodes the video streams using a three-dimensional fully convolutional network, then generates candidate temporal regions containing activities and finally classifies selected regions into specific activities. Computation is saved due to the sharing of convolutional features between the proposal and the classification pipelines. We further improve the detection performance by efficiently integrating an optical flow based motion stream with the original RGB stream. The twostream network is jointly optimized by fusing the flow and RGB feature maps at different levels. Additionally, the training stage incorporates an online hard example mining strategy to address the extreme foreground-background imbalance typically observed in any detection pipeline. Instead of heuristically sampling the candidate segments for the final activity classification stage, we rank them according to their performance and only select the worst performers to update the model. This improves the model without heavy hyper-parameter tuning. Extensive experiments on three benchmark datasets are carried out to show superior performance over existing temporal activity detection methods. Our model achieves state-of-the-art results on the THUMOS'14 and Charades datasets. We further demonstrate that our model is a general temporal activity detection framework that does not rely on assumptions about particular dataset properties by evaluating our approach on the ActivityNet dataset.

show abstract

Section: Fps S-cnn [3]mentioning

confidence: 59%

“…Table VIII provides a comparative evaluation with various baseline models reported in [61]. This approach [61] trains a CRF based video classification model (asynchronous temporal fields) and evaluates the prediction performance on 25 equidistant frames by making a multi-label prediction for each frame.…”

Section: Experiments On Charadesmentioning

confidence: 99%

Two-Stream Region Convolutional 3D Network for Temporal Activity Detection

Das

Saenko

2019

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…For example, on HMDB-51, our results are about 2-3% better than the next best method without IDT-FV. On Charades, we outperform previous methods by about 3% while faring well on the detection task against [81]. We also demonstrate significant performance (about 3-4%) improvement on NTU-RGBD and marginally better performance on MSR datasets on skeleton-based action recognition.…”

Section: Comparisons To the State Of The Artmentioning

confidence: 66%

“…Following the evaluation protocol in [25], we use the output probability of the classifier to be the score of the sequence. In the detection task, we consider the evaluation method with post-processing proposed in [81], which uses the averaged prediction score of a temporal window around each temporal pivots. Instead of average pooling, we apply the SVMP.…”

Section: Action Recognition/detection In Untrimmed Videosmentioning

confidence: 99%

Discriminative Video Representation Learning Using Support Vector Classifiers

Wang

Cherian²

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Most popular deep models for action recognition in videos generate independent predictions for short clips, which are then pooled heuristically to assign an action label to the full video segment. As not all frames may characterize the underlying action-indeed, many are common across multiple actions-pooling schemes that impose equal importance on all frames might be unfavorable. In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action. To identify these useful features, we resort to a negative bag consisting of features that are known to be irrelevant, for example, they are sampled either from datasets that are unrelated to our actions of interest or are CNN features produced via random noise as input. With the features from the video as a positive bag and the irrelevant features as the negative bag, we cast an objective to learn a (nonlinear) hyperplane that separates the unknown useful features from the rest in a multiple instance learning formulation within a support vector machine setup. We use the parameters of this separating hyperplane as a descriptor for the full video segment. Since these parameters are directly related to the support vectors in a max-margin framework, they can be treated as a weighted average pooling of the features from the bags, with zero weights given to non-support vectors. Our pooling scheme is end-to-end trainable within a deep learning framework. We report results from experiments on eight computer vision benchmark datasets spanning a variety of video-related tasks and demonstrate state-of-the-art performance across these tasks.Index Terms-video representation, video data mining, discriminative pooling, action recognition, deep learning. ✦• Jue Wang is with the

show abstract

“…• Asyn-TF (CVPR 2017) [27]: The model attempts to reason over various aspects of activity that includes objects, actions, and intentions.…”

Section: Compared Methodsmentioning

confidence: 99%

Temporal Reasoning Graph for Activity Recognition

Zhang

Shen

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Despite great success has been achieved in activity analysis, it still has many challenges. Most existing work in activity recognition pay more attention to design efficient architecture or video sampling strategy. However, due to the property of fine-grained action and long term structure in video, activity recognition is expected to reason temporal relation between video sequences. In this paper, we propose an efficient temporal reasoning graph (TRG) to simultaneously capture the appearance features and temporal relation between video sequences at multiple time scales. Specifically, we construct learnable temporal relation graphs to explore temporal relation on the multi-scale range. Additionally, to facilitate multi-scale temporal relation extraction, we design a multihead temporal adjacent matrix to represent multi-kinds of temporal relations. Eventually, a multi-head temporal relation aggregator is proposed to extract the semantic meaning of those features convolving through the graphs. Extensive experiments are performed on widely-used large-scale datasets, such as Something-Something and Charades, and the results show that our model can achieve state-of-the-art performance. Further analysis shows that temporal relation reasoning with our TRG can extract discriminative features for activity recognition.

show abstract

Asynchronous Temporal Fields for Action Recognition

Cited by 138 publications

References 59 publications

Two-Stream Region Convolutional 3D Network for Temporal Activity Detection

Two-Stream Region Convolutional 3D Network for Temporal Activity Detection

Discriminative Video Representation Learning Using Support Vector Classifiers

Temporal Reasoning Graph for Activity Recognition

Contact Info

Product

Resources

About