Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

Wray, Michael; Csurka, Gabriela; Larlus, Diane; Damen, Dima

doi:10.1109/iccv.2019.00054

Cited by 147 publications

(108 citation statements)

References 35 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…Kazakos et al [69] proposed to fuse video and audio signals for FPV action recognition. Wray et al [70] made use of text descriptions of FPV actions for zero-short learning. In contrast to our work, these prior work did not consider using egocentric gaze for action recognition.…”

Section: First Person Visionmentioning

confidence: 99%

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Liu

Rehg

2018

Lecture Notes in Computer Science

244

329

View full text Add to dashboard Cite

We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action annotations, thereby providing the most comprehensive benchmark for First Person Vision (FPV). Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We further sample from these stochastic units, generating an attention map to guide the aggregation of visual features for action recognition. Our method is evaluated on our EGTEA Gaze+ dataset and achieves a performance level that exceeds the state-of-the-art by a significant margin. More importantly, we demonstrate that our model can be applied to larger scale FPV dataset-EPIC-Kitchens even without using gaze, offering new state-of-the-art results on FPV action recognition.

show abstract

Section: First Person Visionmentioning

confidence: 99%

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Liu

Rehg

2018

Lecture Notes in Computer Science

244

329

View full text Add to dashboard Cite

show abstract

“…Specifically, our model achieves 8.9 R@1 improvement over the original HowTo100M model (Miech et al, 2019) and other recent baselines with pre-training on HowTo100M. Using a smaller set of visual fea-Model R@1↑ R@5↑ R@10↑ JSFusion 10.2 31.2 43.2 JPoSE (Wray et al, 2019) 14. English to Video Chinese to Video Model R@1↑ R@5↑ R10↑ R@1↑ R@5↑ R@10↑ VSE (Kiros et al, 2014) 28.0 64.3 76.9 ---VSE++ (Faghri et al, 2018) 33.7 70.1 81.0 ---Dual (Dong et al, 2019) 31.1 67.4 78.9 ---HGR (Chen et al, 2020a) 35.1 73.5 83.…”

Section: Comparison To Supervised State Of the Artmentioning

confidence: 88%

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Huang¹,

Patrick²,

Hu³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-tovideo search and propose a Transformer-based model that learns contextual multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K.

show abstract

“…Each epoch training is just performed using a single GPU and takes no more than 10 minutes. [30] , MEE [27], MMEN [43], and JPoSE [43], and (3) other methods: JSFusion [49], CCA (FV HGLMM) [16], and Miech et al [26]. The experimental results on MSR-VTT and LSMDC are summarized, respectively, in Table 1 and Table 2.…”

Section: Methodsmentioning

confidence: 99%

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

Yang

Dong

Cao

et al. 2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

121

View full text Add to dashboard Cite

The rapid growth of user-generated videos on the Internet has intensified the need for text-based video retrieval systems. Traditional methods mainly favor the concept-based paradigm on retrieval with simple queries, which are usually ineffective for complex queries that carry far more complex semantics. Recently, embedding-based paradigm has emerged as a popular approach. It aims to map the queries and videos into a shared embedding space where semantically-similar texts and videos are much closer to each other. Despite its simplicity, it forgoes the exploitation of the syntactic structure of text queries, making it suboptimal to model the complex queries. To facilitate video retrieval with complex queries, we propose a Tree-augmented Cross-modal Encoding method by jointly learning the linguistic structure of queries and the temporal representation of videos. Specifically, given a complex user query, we first recursively compose a latent semantic tree to structurally describe the text query. We then design a tree-augmented query encoder to derive structure-aware query representation and a temporal attentive video encoder to model the temporal characteristics of videos. Finally, both the query and videos are mapped into a joint embedding space for matching and ranking. In this approach, we have a better understanding and modeling of the complex queries, thereby achieving a better video retrieval performance. Extensive experiments on large scale video retrieval benchmark datasets demonstrate the effectiveness of our approach. CCS CONCEPTS • Information systems → Multimedia and multimodal retrieval; Video search.

show abstract

Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings

Cited by 147 publications

References 35 publications

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

Contact Info

Product

Resources

About