Semantic text-based image retrieval with multi-modality ontology and DBpedia

Abstract

Purpose The purpose of this study is to reduce the semantic distance by proposing a model for integrating indexes of textual and visual features via a multi-modality ontology and the use of DBpedia to improve the comprehensiveness of the ontology to enhance semantic retrieval. Design/methodology/approach A multi-modality ontology-based approach was developed to integrate high-level concepts and low-level features, as well as integrate the ontology base with DBpedia to enrich the knowledge resource. A complete ontology model was also developed to represent the domain of sport news, with image caption keywords and image features. Precision and recall were used as metrics to evaluate the effectiveness of the multi-modality approach, and the outputs were compared with those obtained using a single-modality approach (i.e. textual ontology and visual ontology). Findings The results based on ten queries show a superior performance of the multi-modality ontology-based IMR system integrated with DBpedia in retrieving correct images in accordance with user queries. The system achieved 100 per cent precision for six of the queries and greater than 80 per cent precision for the other four queries. The text-based system only achieved 100 per cent precision for one query; all other queries yielded precision rates less than 0.500. Research limitations/implications This study only focused on BBC Sport News collection in the year 2009. Practical implications The paper includes implications for the development of ontology-based retrieval on image collection. Originality value This study demonstrates the strength of using a multi-modality ontology integrated with DBpedia for image retrieval to overcome the deficiencies of text-based and ontology-based systems. The result validates semantic text-based with multi-modality ontology and DBpedia as a useful model to reduce the semantic distance.

Constructing a mobile visual search framework for Dunhuang murals based on fine-tuned CNN and ontology semantic distance

“…Ramli et al (2016) proposed an ontology-based approach to index and rank semantically rich historical documents. Multi-modality ontology and DBpedia are used for semantic TBIR (M.K. and Noah, 2017).…”

A Semantic Image Retrieval Technique Through Concept Co-occurrence Based Database Organization and DeepLab Segmentation

“…This prevents the problem of overfitting in traditional convolutional networks and improves accuracy. The application of deep learning for region retrieval is presented by several authors (Gordo et al, 2016;Liu et al 2017;Yanti Idaya Aspura and Mohd Noah, 2017) where CNN features are used for ROI representation. The relevance and drawbacks of the above mentioned retrieval works are listed out in Table 1 for a quick understanding.…”