Cross-Modal Object Detection Based on a Knowledge Update

Abstract

As an important field of computer vision, object detection has been studied extensively in recent years. However, existing object detection methods merely utilize the visual information of the image and fail to mine the high-level semantic information of the object, which leads to great limitations. To take full advantage of multi-source information, a knowledge update-based multimodal object recognition model is proposed in this paper. Specifically, our method initially uses Faster R-CNN to regionalize the image, then applies a transformer-based multimodal encoder to encode visual region features (region-based image features) and textual features (semantic relationships between words) corresponding to pictures. After that, a graph convolutional network (GCN) inference module is introduced to establish a relational network in which the points denote visual and textual region features, and the edges represent their relationships. In addition, based on an external knowledge base, our method further enhances the region-based relationship expression capability through a knowledge update module. In summary, the proposed algorithm not only learns the accurate relationship between objects in different regions of the image, but also benefits from the knowledge update through an external relational database. Experimental results verify the effectiveness of the proposed knowledge update module and the independent reasoning ability of our model.

Hybrid DAER Based Cross-Modal Retrieval Exploiting Deep Representation Learning

“…Several studies have developed diverse deep cross-modal representation learning methods that utilize the neural networks to extract image and text features and build relationships between different modalities. For example, Gao et al [ 27 ] proposed an image encoder, text encoder, and multi-modal encoder to extract text features and image features and mine rich feature information. Viviana et al [ 28 ] used convolutional neural networks to deeply extract features from images and text in the stage of establishing connections between different modal data.…”

Substation instrumentation target detection based on multi‐scale feature fusion

“…Target detection technology has a very important role in the field of machine vision and practical applications in life, so the automatic detection of targets is an important research task 24–27 . The performance of target detection can be seriously affected by complex background, target occlusion, noise interference, low resolution, scale and pose variation and so forth 28 . Among the many instruments in substations, the appearance and expression of instruments with different functions are very different, but there are similar characteristics for different categories of industrial instruments 29–32 .…”

“…[24][25][26][27] The performance of target detection can be seriously affected by complex background, target occlusion, noise interference, low resolution, scale and pose variation and so forth. 28 Among the many instruments in substations, the appearance and expression of instruments with different functions are very different, but there are similar characteristics for different categories of industrial instruments. [29][30][31][32] At the same time, when acquiring the image information of substation meters, the different acquisition methods of substation inspection robots will also lead to some different types of interference information, such as picture noise, which will affect the accuracy of detection results to a certain extent.…”