Knowledge Graph Representation Fusion Framework for Fine‐Grained Object Recognition in Smart Cities

He, Yang; Tian, Ling; Zhang, Lizong; Xi, Zhenfeng

doi:10.1155/2021/8041029

Cited by 6 publications

(2 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to aid in the recognition of images [38,39], classification hierarchies such as WordNet [40] have often been utilized. Knowledge graphs have also been widely used for visual classification and recognition [41,42].…”

Section: Application Of Knowledge In Computer Visionmentioning

confidence: 99%

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Cheng,

Jiang,

et al. 2024

Preprint

View full text Add to dashboard Cite

Current works in Visual Question Answering(VQA) that introduce external knowledge mainly focus on leveraging the knowledge to supplement the language representation in VQA model’s question input. However, such approaches ignore the commonsense information implied in the image. In this paper, we propose a novel VQA framework to embed knowledge features into vision and language respectively with a shared knowledge graph. To bridge the gap between visual representation and knowledge representation, we propose the knowledge enhancing visual representation (KEVR) module. KEVR is designed to explore external knowledge in image from the knowledge graph. By utilizing KEVR, external knowledge related to the objects in the image can be embedded into visual representation directly. In terms of the input question, a designed transformer is utilized to embed knowledge features into language representation. The knowledge graph used in our model is extracted from three knowledge bases. We organize the prior knowledge in the form of RDF triples to establish knowledge connections, then a graph neural network is employed to extract multilateral relationships in knowledge graph. What's more, a two-stream transformer is employed to get the attention based vision-language representation. The results of our experiments show that our model outperforms the best baseline model with an accuracy of 1.34% and 2.59% on VQA 2.0 and OK-VQA datasets respectively.

show abstract

Section: Application Of Knowledge In Computer Visionmentioning

confidence: 99%

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Cheng,

Jiang,

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Zheng et al [58] obtain knowledge graphs by extracting abundant visual concepts and then combining DNN structures with professional knowledge for scene understanding. To achieve fine-grained image classification, He et al [59] propose a Knowledge Graph Representation Fusion (KGRF) framework by using prior knowledge. Besides, Rambhatla et al [55] propose a working and semantic memory framework to discover unknown categories when prior knowledge is known.…”

Section: Knowledge Based Visual Related Tasksmentioning

confidence: 99%

Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding

Wu¹,

Ji²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this paper, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the highspeed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this paper, to tackle the dilemma of speedaccuracy tradeoff, we propose a commonsense-aware cross-modal alignment (CCA) framework, which incorporates commonsenseguided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our CCA method performs favorably against state-ofthe-arts while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA.

show abstract

Vision and Knowledge Bimodal Fine-grained Recognition Based on Hierarchical Structured Knowledge Extraction

Zou

Zhu

Chen

et al. 2022

2022 4th International Conference on Data Intelligence and Security (ICDIS)

View full text Add to dashboard Cite

Knowledge Graph Representation Fusion Framework for Fine‐Grained Object Recognition in Smart Cities

Cited by 6 publications

References 52 publications

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding

Vision and Knowledge Bimodal Fine-grained Recognition Based on Hierarchical Structured Knowledge Extraction

Contact Info

Product

Resources

About