Multimodal Query-Guided Object Localization

Tripathi, Aditay; Dani, Rajath R; Mishra, Anand; Chakraborty, Anirban

doi:10.2139/ssrn.3989558

Cited by 1 publication

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, efforts have been made towards recognizing individuals based on their facial features (Sivic, Everingham, and Zisserman 2005) or clothing characteristics (Brunelli and Falavigna 1995). In a more novel direction, some research has explored using natural language or sketch-based queries to precisely locate objects within images (Minderer et al 2022;Sadhu, Chen, and Nevatia 2019;Kumar and Mishra 2023;Tripathi et al 2020Tripathi et al , 2023. As visual comprehension continues to advance, there is a need for localization methodologies that can effectively Copyright © 2024, Association for the Advancement of Artificial Intelligence (www.aaai.org).…”

Section: Introductionmentioning

confidence: 99%

QDETRv: Query-Guided DETR for One-Shot Object Localization in Videos

Kumar,

Mallick,

Mishra

et al. 2024

AAAI

View full text Add to dashboard Cite

In this work, we study one-shot video object localization problem that aims to localize instances of unseen objects in the target video using a single query image of the object. Toward addressing this challenging problem, we extend a popular and successful object detection method, namely DETR (Detection Transformer), and introduce a novel approach –query-guided detection transformer for videos (QDETRv). A distinctive feature of QDETRv is its capacity to exploit information from the query image and spatio-temporal context of the target video, which significantly aids in precisely pinpointing the desired object in the video. We incorporate cross-attention mechanisms that capture temporal relationships across adjacent frames to handle the dynamic context in videos effectively. Further, to ensure strong initialization for QDETRv, we also introduce a novel unsupervised pretraining technique tailored to videos. This involves training our model on synthetic object trajectories with an analogous objective as the query-guided localization task. During this pretraining phase, we incorporate recurrent object queries and loss functions that encourage accurate patch feature reconstruction. These additions enable better temporal understanding and robust representation learning. Our experiments show that the proposed model significantly outperforms the competitive baselines on two public benchmarks, VidOR and ImageNet-VidVRD, extended for one-shot open-set localization tasks.

show abstract