Image classification using multimedia knowledge networks

Abstract

This paper presents novel methods for classifying images based on knowledge discovered from annotated images using WordNet. The novelty of this work is the automatic class discovery and the classifier combination using the extracted knowledge. The extracted knowledge is a network of concepts (e.g., image clusters and word-senses) with associated image and text examples. Concepts that are similar statistically are merged to reduce the size of the concept network. Our knowledge classifier is constructed by training a meta-classifier to predict the presence of each concept in images. A Bayesian network is then learned using the meta-classifiers and the concept network. For a new image, the presence of concepts is first detected using the meta-classifiers and refined using Bayesian inference. Experiments have shown that combining classifiers using knowledge-based Bayesian networks results in superior (up to 15%) or comparable accuracy to individual classifiers and purely statistically learned classifier structures. Another contribution of this work is the analysis of the role of visual and text features in image classification. As text or joint text+visual features perform better in classifying images than visual features, we tried to predict text features for images without annotations; however, the accuracy of visual + predicted text features did not consistently improve over visual features.

Using Visual Context and Region Semantics for High-Level Concept Detection

“…Thus, each feature vector selected to represent each region type is given by (4). The distance between two feature vectors is denoted by (5). These region types are the centroids of the clusters and all the other feature vectors of a cluster are their synonyms.…”

“…, with the feature vectors and Euclidean distance , as the latter is denoted by (5). is the set of feature vectors for the specific set of regions, whereas is the entire feature vector space.…”

“…At this point we must note that the MPEG-7 standard does not specify strict distance measures for the descriptors; it only suggests some, so as to allow for other measures to be used and test their efficiency. As depicted in experiments in our previous work [69], the use of the Euclidean distance provides a simple way to use all extracted low-level information, leading also to satisfactory results (5) Having calculated the distance of each region (cluster) of the image to all the words of the constructed thesaurus, the model vector that semantically describes the visual content of the image is formed by keeping the smaller distance for each intermediate concept (region type). After extracting model vectors from all images of the (annotated) training set, a neural network-based detector is trained separately for each high-level concept.…”

“…2008.2009681 content analysis phase, is a challenging and broad research area for any researcher. Although the well-known "semantic gap" [64] has been acknowledged for a long time, multimedia analysis approaches are still divided into two rather discrete categories; low-level multimedia analysis methods and tools, on the one hand (e.g., [59]) and high-level semantic annotation methods and tools, on the other (e.g., [78], [5]). It was only recently, that state-of-the-art multimedia analysis systems have started using semantic knowledge technologies, as the latter are defined by notions like ontologies [74], folksonomies [43] and the Semantic Web standards.…”

Semantic Multimedia Analysis based on Region Types and Visual Context

“…[13]) and high-level semantic annotation methods and tools, on the other (e.g. [20], [3]). It was only recently, that state-of-the-art multimedia analysis systems have started using semantic knowledge technologies, as the latter are defined by notions like ontologies [19] and whose advantages, when using them for the creation, manipulation and post-processing of multimedia metadata, are depicted in numerous research activities.…”