ER3: A Unified Framework for Event Retrieval, Recognition and Recounting

Gao, Zhanning; Hua, Gang; Zhang, Dongqing; Jojić, Nebojša; Wang, Le; Xue, Jianru; Zheng, Nanning

doi:10.1109/cvpr.2017.227

Cited by 26 publications

(23 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we apply our approach on event retrieval. We compare against the Mean-MultiVLAD (MMV), obtained by averaging and 2 -normalizing Multi-VLAD frame descriptors, CTE [28], Stable hyper-pooling [6] and the recent Counting Grid Aggregation (CGA) [11]. LAMV, CTE and TMK are able to provide a good localization in addition to retrieval, the others can not.…”

Section: Resultsmentioning

confidence: 99%

“…According to a comparative study on copy detection conducted in 2014 [19], the best methods were relying on local descriptors and framebased matching [18], even though temporal alignment is often needed later, for example to manually verify a copyright infringement. In contrast, the state of the art for particular event retrieval [6,11] exploits a single vector per video.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LAMV: Learning to Align and Match Videos with Kernelized Temporal Layers

Baraldi

Douze

Cucchiara

2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

This paper considers a learnable approach for comparing and aligning videos. Our architecture builds upon and revisits temporal match kernels within neural networks: we propose a new temporal layer that finds temporal alignments by maximizing the scores between two sequences of vectors, according to a time-sensitive similarity metric parametrized in the Fourier domain. We learn this layer with a temporal proposal strategy, in which we minimize a triplet loss that takes into account both the localization accuracy and the recognition rate. We evaluate our approach on video alignment, copy detection and event retrieval. Our approach outperforms the state on the art on temporal video alignment and video copy detection datasets in comparable setups. It also attains the best reported results for particular event search, while precisely aligning videos. * This work was carried out while L. Baraldi was in a internship at Facebook AI Research. This academic article may contain images and/or data from sources that are not affiliated with the article submitter. Inclusion should not be construed as approval, endorsement or sponsorship of the submitter, article or its content by any such party.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

LAMV: Learning to Align and Match Videos with Kernelized Temporal Layers

Baraldi

Douze

Cucchiara

2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…A straightforward approach to this is to aggregate/pool frame-level features into a single video-level representation on which subsequently one can calculate a similarity measure. Such video-level representations include global vectors [35,11,21], hash codes [30,23,31] and Bag-of-Words (BoW) [5,20,22]. However, this disregards the spatial and the temporal structure of the visual similarity, as aggregation of features is influenced by clutter and irrelevant content.…”

Section: Introductionmentioning

confidence: 99%

ViSiL: Fine-Grained Spatio-Temporal Video Similarity Learning

Kordopatis-Zilos

Papadopoulos

Patras

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra-and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features -this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available 1 .

show abstract

“…A simpler alternative is proposed in this paper as the "temporal Detection By spatial Segmentation" (DBS) framework, which circumvents the variable lengths input difficulty. As illustrated in Figure 1, temporal action detection is recast into spatial semantic segmentation with the video imprint representation (Gao et al 2017b;2018), by aligning video frames into a fixed-size tensor feature. Such representation captures statistical characteristics while suppressing redundancies.…”

Section: Introductionmentioning

confidence: 99%

“…Then, Figure 1: Overview of the temporal detection by spatial segmentation framework for temporal action detection. Based on the video imprint (Gao et al 2017b;2018), video frames are nonlinearly projected into the video imprint, as visualized at the bottom left "black cube". FCN-based network is utilized to segment the imprint representation and the corresponding prediction score maps are visualized as a "blue cube" at the bottom center.…”

Section: Introductionmentioning

confidence: 99%

Video Imprint Segmentation for Temporal Action Detection in Untrimmed Videos

Gao

Wang

Zhang³

et al. 2019

AAAI

Self Cite

View full text Add to dashboard Cite

We propose a temporal action detection by spatial segmentation framework, which simultaneously categorize actions and temporally localize action instances in untrimmed videos. The core idea is the conversion of temporal detection task into a spatial semantic segmentation task. Firstly, the video imprint representation is employed to capture the spatial/temporal interdependences within/among frames and represent them as spatial proximity in a feature space. Subsequently, the obtained imprint representation is spatially segmented by a fully convolutional network. With such segmentation labels projected back to the video space, both temporal action boundary localization and per-frame spatial annotation can be obtained simultaneously. The proposed framework is robust to variable lengths of untrimmed videos, due to the underlying fixed-size imprint representations. The efficacy of the framework is validated in two public action detection datasets.

show abstract

ER3: A Unified Framework for Event Retrieval, Recognition and Recounting

Cited by 26 publications

References 37 publications

LAMV: Learning to Align and Match Videos with Kernelized Temporal Layers

LAMV: Learning to Align and Match Videos with Kernelized Temporal Layers

ViSiL: Fine-Grained Spatio-Temporal Video Similarity Learning

Video Imprint Segmentation for Temporal Action Detection in Untrimmed Videos

Contact Info

Product

Resources

About