FindIt: Generalized Localization with Natural Language Queries

Kuo, Wen Chuan; Bertsch, Fred; Li, Wei; Piergiovanni, AJ; Saffar, Mohammad Hajizadeh; Angelova, Anelia

doi:10.1007/978-3-031-20059-5_29

Cited by 21 publications

(30 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bansal et al [4] introduce ZS+OV detection where the classification layer of a closed vocabulary detector is replaced with the text embeddings of the class names, an approach taken by many subsequent works [11,14,16,24,31,42,46,46], including this one. Some works [16,24,42] take the OV classification closer to the backbone features by directly extracting them from object proposals with ROI-Align [20], and optionally distill a strong OV classifier into the detector [16]. To improve ZS performance, Detic [46] and PromptDet [14] forego the OV aspect -knowing the names of the classes of interest (i.e.…”

Section: Related Workmentioning

confidence: 99%

“…We use the LVIS v1.0 [18] object detection benchmark adapted for zero-shot evaluation; we call this setup LVIS -R . Following standard practice [16,24,46], the rare class annotations are removed from the training set, keeping only the frequent and common annotations (often called LVIS-base). Evaluation is then performed on all classes, reporting the box mAP for all classes (mAP all ) and the mAP on rare classes (mAP rare ), with the emphasis on mAP rare as this measures the zero-shot performance, rare classes playing the role of the unseen classes.…”

Section: Baseline Detector and Experimental Setupmentioning

confidence: 99%

“…(i) Many works [24,42,46] choose to freeze the pretrained language model, while others, observe this yielding bad performance [26,31,43] and choose to train it, but with a small learning rate to prevent "catastrophic forgetting". We also find that a frozen language model alone yields poor performance (Section 3.1.1), but propose instead to use the frozen language model together with a simple and efficient data augmentation approach, which provides superior results to both alternatives while speeding up training and decreasing accelerator memory consumption.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Three ways to improve feature alignment for open vocabulary detection

Arandjelović¹,

Andonian²,

Arthur³

et al. 2023

Preprint

View full text Add to dashboard Cite

The core problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes. Previous approaches train the feature pyramid and detection head from scratch, which breaks the vision-text feature alignment established during pretraining, and struggles to prevent the language model from forgetting unseen classes.We propose three methods to alleviate these issues. Firstly, a simple scheme is used to augment the text embeddings which prevents overfitting to a small number of classes seen during training, while simultaneously saving memory and computation. Secondly, the feature pyramid network and the detection head are modified to include trainable gated shortcuts, which encourages vision-text feature alignment and guarantees it at the start of detection training. Finally, a self-training approach is used to leverage a larger corpus of image-text pairs thus improving detection performance on classes with no human annotated bounding boxes.Our three methods are evaluated on the zero-shot version of the LVIS benchmark, each of them showing clear and significant benefits. Our final network achieves the new stateof-the-art on the mAP-all metric and demonstrates competitive performance for mAP-rare, as well as superior transfer to COCO and Objects365. * Equal contribution. • Work done during internship at DeepMind.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Baseline Detector and Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Three ways to improve feature alignment for open vocabulary detection

Arandjelović¹,

Andonian²,

Arthur³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Our approach combines visual and language modalities to improve object anticipation. Most recent multimodal architectures are based on Transformer fusion schemes applied on features extracted by various encoders [5,12,27]. We follow the spirit by employing self-attention-based modality fusion.…”

Section: Related Workmentioning

confidence: 99%

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction

Pasca¹,

Gavryushin²,

Kuo³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study the task of object interaction anticipation in egocentric videos. Successful prediction of future actions and objects requires an understanding of the spatiotemporal context formed by past actions and object relationships.We propose TransFusion, a multimodal transformer-based architecture, that effectively makes use of the representational power of language by summarizing past actions concisely. TransFusion leverages pre-trained image captioning models and summarizes the caption, focusing on past actions and objects. This action context together with a single input frame is processed by a multimodal fusion module to forecast the next object interactions. Our model enables more efficient end-to-end learning by replacing dense video features with language representations, allowing us to benefit from knowledge encoded in large pre-trained models. Experiments on Ego4D and EPIC-KITCHENS-100 show the effectiveness of our multimodal fusion model and the benefits of using language-based context summaries. Our method outperforms state-of-the-art approaches by 40.4% in overall mAP on the Ego4D test set. We show the generality of TransFusion via experiments on EPIC-KITCHENS-100. Video and code are available at: https://eth-ait.github.io/transfusion-proj/.

show abstract

“…FIBER (Dou et al, 2022a) improves GLIP by (i) using a coarse-to-fine pretraining pipeline, and (ii) performing fusion in the backbone rather than in the OD head as in GLIP. (Zang et al, 2022), X-DETR (Cai et al, 2022), FindIT (Kuo et al, 2022), PromptDet (Feng et al, 2022), and OWL-ViT (Minderer et al, 2022).…”

Section: Two-stage Modelsmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

FindIt: Generalized Localization with Natural Language Queries

Cited by 21 publications

References 72 publications

Three ways to improve feature alignment for open vocabulary detection

Three ways to improve feature alignment for open vocabulary detection

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Contact Info

Product

Resources

About