Mixture-Kernel Graph Attention Network for Situation Recognition

Suhail, Mohammed; Sigal, Leonid

doi:10.1109/iccv.2019.01046

Cited by 24 publications

(17 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For ImSitu actions we prompt the model with "What are they doing?". GPV-2 gets 34.7 top-5 accuracy compared to 58.6 from the benchmark authors [87] employing a supervised CNN+CRF approach and 68.6 from a recent supervised model [72] that uses a specialized mixturekernel attention graph neural network. For verbs present in WEB10K (the Seen column), WEB10K training provides a significant boost (54.4 from 33.4) showing successful transfer from web images to ImSitu images.…”

Section: Methodsmentioning

confidence: 99%

Webly Supervised Concept Expansion for General Purpose Vision Models

Kamath¹,

Clark²,

Gupta³

et al. 2022

Preprint

View full text Add to dashboard Cite

General purpose vision (GPV) systems [25] are models that are designed to solve a wide array of visual tasks without requiring architectural changes. Today, GPVs primarily learn both skills and concepts from large fully supervised datasets. Scaling GPVs to tens of thousands of concepts by acquiring data to learn each concept for every skill quickly becomes prohibitive. This work presents an effective and inexpensive alternative: learn skills from fully supervised datasets, learn concepts from web image search results, and leverage a key characteristic of GPVs -the ability to transfer visual knowledge across skills. We use a dataset of 1M+ images spanning 10k+ visual concepts to demonstrate webly-supervised concept expansion for two existing GPVs (GPV-1 [25] and VL-T5 [14]) on 3 benchmarks -5 COCO based datasets (80 primary concepts), a newly curated series of 5 datasets based on the OpenImages and VisualGenome repositories (∼500 concepts) and the Web-derived dataset (10k+ concepts). We also propose a new architecture, GPV-2 that supports a variety of tasks -from vision tasks like classification and localization to vision+language tasks like QA and captioning to more niche ones like human-object interaction recognition. GPV-2 benefits hugely from web data, outperforms GPV-1 and VL-T5 across these benchmarks, and does well in a 0-shot setting at action and attribute recognition.

show abstract

Section: Methodsmentioning

confidence: 99%

Webly Supervised Concept Expansion for General Purpose Vision Models

Kamath¹,

Clark²,

Gupta³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The common pipeline of SR and GSR in the literature [3,4,19,27,29,32,40,41] resembles the two processes: predicting a verb (Glance), then estimating a noun for each role associated with the predicted verb (Gaze). Regarding this pipeline, correctness of the predicted verb is extremely important since noun estimation entirely depends on the predicted verb.…”

Section: Glance Gazementioning

confidence: 99%

Collaborative Transformers for Grounded Situation Recognition

Cho¹,

Kwak²

2022

Preprint

View full text Add to dashboard Cite

Grounded situation recognition is the task of predicting the main activity, entities playing certain roles within the activity, and bounding-box groundings of the entities in the given image. To effectively deal with this challenging task, we introduce a novel approach where the two processes for activity classification and entity estimation are interactive and complementary. To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation. Glance transformer predicts the main activity with the help of Gaze transformer that analyzes entities and their relations, while Gaze transformer estimates the grounded entities by focusing only on the entities relevant to the activity predicted by Glance transformer. Our CoFormer achieves the state of the art in all evaluation metrics on the SWiG dataset. Training code and model weights are available at https://github.com/jhcho99/CoFormer.

show abstract

“…Inspired by image captioning task, Mallya and Lazebnik [15] adopt a Recurrent Neural Network (RNN) architecture to model the relations in the predefined order. Li et al [9] use a Gated Graph Neural Network (GGNN) [10] to capture relations among roles, and Suhail and Sigal [20] propose a modified GGNN to learn context-aware relations among roles depending on the content of the image. Cooray et al [2] formulate the relation modeling as an interdependent query based visual reasoning problem.…”

Section: Related Workmentioning

confidence: 99%

Grounded Situation Recognition with Transformers

Cho

Yoon

Lee

et al. 2021

Preprint

View full text Add to dashboard Cite

Grounded Situation Recognition (GSR) is the task that not only classifies a salient action (verb), but also predicts entities (nouns) associated with semantic roles and their locations in the given image. Inspired by the remarkable success of Transformers in vision tasks, we propose a GSR model based on a Transformer encoder-decoder architecture. The attention mechanism of our model enables accurate verb classification by capturing high-level semantic feature of an image effectively, and allows the model to flexibly deal with the complicated and image-dependent relations between entities for improved noun classification and localization. Our model is the first Transformer architecture for GSR, and achieves the state of the art in every evaluation metric on the SWiG benchmark. Our code is available at https://github.com/jhcho99/gsrtr.

show abstract

Mixture-Kernel Graph Attention Network for Situation Recognition

Cited by 24 publications

References 19 publications

Webly Supervised Concept Expansion for General Purpose Vision Models

Webly Supervised Concept Expansion for General Purpose Vision Models

Collaborative Transformers for Grounded Situation Recognition

Grounded Situation Recognition with Transformers

Contact Info

Product

Resources

About