Spatio-Temporal Video Re-Localization by Warp LSTM

Feng, Yang; Ma, Lin; Liu, Wei; Luo, Jiebo

doi:10.1109/cvpr.2019.00138

Cited by 39 publications

(35 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ing the task of video re-localization in diverse ways. For example, Feng et al (Feng et al 2019) not only localize the temporal boundary of an action clip but also seek the spatial bounding box of that action at each frame. Zhang et al (Zhang et al 2019b) try to use a single query image to localize the relevant action clip in a reference video.…”

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Video Re-Localization with Multiscale Attention Model

Huang

Hsu

Jeng

et al. 2020

AAAI

View full text Add to dashboard Cite

Video re-localization aims to localize a sub-sequence, called target segment, in an untrimmed reference video that is similar to a given query video. In this work, we propose an attention-based model to accomplish this task in a weakly supervised setting. Namely, we derive our CNN-based model without using the annotated locations of the target segments in reference videos. Our model contains three modules. First, it employs a pre-trained C3D network for feature extraction. Second, we design an attention mechanism to extract multiscale temporal features, which are then used to estimate the similarity between the query video and a reference video. Third, a localization layer detects where the target segment is in the reference video by determining whether each frame in the reference video is consistent with the query video. The resultant CNN model is derived based on the proposed co-attention loss which discriminatively separates the target segment from the reference video. This loss maximizes the similarity between the query video and the target segment while minimizing the similarity between the target segment and the rest of the reference video. Our model can be modified to fully supervised re-localization. Our method is evaluated on a public dataset and achieves the state-of-the-art performance under both weakly supervised and fully supervised settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Video Re-Localization with Multiscale Attention Model

Huang

Hsu

Jeng

et al. 2020

AAAI

View full text Add to dashboard Cite

show abstract

“…To benefit from the space context, spatial pooling divides a video using fixed segmentation grids and pools the features locally in each grid cell [22][23][24]. Though the performance has been improved, different action instances of the same category with various human localizations in spatiotemporal volume [46,47] can result in a non-uniform distribution of features. Furthermore, one interaction component may be divided into different cells due to the fixed segmentation.…”

Section: Mid-level Patch Mining Based On Motion Saliencymentioning

confidence: 99%

Fine-Grained Action Recognition by Motion Saliency and Mid-Level Patches

et al. 2020

View full text Add to dashboard Cite

Effective extraction of human body parts and operated objects participating in action is the key issue of fine-grained action recognition. However, most of the existing methods require intensive manual annotation to train the detectors of these interaction components. In this paper, we represent videos by mid-level patches to avoid the manual annotation, where each patch corresponds to an action-related interaction component. In order to capture mid-level patches more exactly and rapidly, candidate motion regions are extracted by motion saliency. Firstly, the motion regions containing interaction components are segmented by a threshold adaptively calculated according to the saliency histogram of the motion saliency map. Secondly, we introduce a mid-level patch mining algorithm for interaction component detection, with object proposal generation and mid-level patch detection. The object proposal generation algorithm is used to obtain multi-granularity object proposals inspired by the idea of the Huffman algorithm. Based on these object proposals, the mid-level patch detectors are trained by K-means clustering and SVM. Finally, we build a fine-grained action recognition model using a graph structure to describe relationships between the mid-level patches. To recognize actions, the proposed model calculates the appearance and motion features of mid-level patches and the binary motion cooperation relationships between adjacent patches in the graph. Extensive experiments on the MPII cooking database demonstrate that the proposed method gains better results on fine-grained action recognition.

show abstract

“…However, to be effective they need to generate many proposals and a secondary supervised step to find the best fitting one. To avoid the need for extensive supervision, Feng et al [7] introduced one-shot localization of actions in time and space. They rely on proposals as well, but rather than using class supervision, a matching model between a trimmed support video and a long untrimmed video determines the best proposal.…”

Section: Introductionmentioning

confidence: 99%

“…upon both [7] and [48] and propose the new task of few-shot common action localization in time and space. Our approach does not require any box annotations or class labels to obtain the spatio-temporal localization, and neither do we need proposals as in [7,48]. All we require are a handful of trimmed videos showing a common unnamed action, see Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Few-Shot Transformation of Common Actions into Time and Space

Yang

Mettes

Snoek

2021

Preprint

View full text Add to dashboard Cite

This paper introduces the task of few-shot common action localization in time and space. Given a few trimmed support videos containing the same but unknown action, we strive for spatio-temporal localization of that action in a long untrimmed query video. We do not require any class labels, interval bounds, or bounding boxes. To address this challenging task, we introduce a novel few-shot transformer architecture with a dedicated encoder-decoder structure optimized for joint commonality learning and localization prediction, without the need for proposals. Experiments on our reorganizations of the AVA and UCF101-24 datasets show the effectiveness of our approach for few-shot common action localization, even when the support videos are noisy. Although we are not specifically designed for common localization in time only, we also compare favorably against the few-shot and one-shot state-of-the-art in this setting. Lastly, we demonstrate that the few-shot transformer is easily extended to common action localization per pixel.

show abstract

Spatio-Temporal Video Re-Localization by Warp LSTM

Cited by 39 publications

References 47 publications

Weakly-Supervised Video Re-Localization with Multiscale Attention Model

Weakly-Supervised Video Re-Localization with Multiscale Attention Model

Fine-Grained Action Recognition by Motion Saliency and Mid-Level Patches

Few-Shot Transformation of Common Actions into Time and Space

Contact Info

Product

Resources

About