Automatic image annotation and retrieval using cross-media relevance models

Jeon, Jiwoon; Lavrenko, Victor; Manmatha, R.

doi:10.1145/860435.860459

Cited by 823 publications

(478 citation statements)

References 19 publications

Supporting

Mentioning

470

Contrasting

Unclassified

Order By: Relevance

“…In general, researchers have proposed knowledge modeling methods for automatic annotation, such as classification-based methods [2,4,15,23], graphical model-based methods [36,37], cross-media modeling methods [9,11,17], and translation modelbased [5,7,12] methods. The key attributes of these methods are using different machine learning algorithms on a training set and the construction of mapping relationships between the semantic concepts and low-level features.…”

Section: Related Workmentioning

confidence: 99%

High-level semantic image annotation based on hot Internet topics

Wang

et al. 2013

Multimed Tools Appl

View full text Add to dashboard Cite

Images are complex multimedia data that contain rich semantic information. Currently, most of image annotation algorithms are only annotating the object semantics of images. There are still many challenges on high-level semantic image annotation. The major issues are the lack of effective modeling method for the highlevel semantics of images and the lack of efficient dynamic update mechanism for the training set. To address these issues, we propose a high-level semantic annotation method based on hot Internet topics in this paper. There are two independent sub tasks in our method: dynamic update of the training set based on hot Internet topics and search-based image annotation. In the first sub task, we propose to model the abstract semantics of images based on three relationships: image-to-image similarity relationship, topic-to-topic co-occurrence relationship, and image-to-topic relevance relationship. Through the complex graph clustering, the hot Internet topics are extracted for images with consistent visual and semantic contents. Then the dynamic update mechanism will update the original training set with the new topics and images. It avoids the huge computing cost in traditional update methods and does not need to re-calculate the whole mapping relationship between the semantic concepts and visual features. In the second sub task, given a query image, it first searches for similar candidates in the annotated training set via visual features. Then the hypergraph modeling and spectral clustering are exploited to filter out the images with irrelevant semantics. The keywords will be extracted for annotation from the remaining images according to an annotation probability. Extensive experiments have been conducted and the results demonstrate that our algorithm could achieve better annotation performance than the state-of-the-art algorithms. And the update mechanism could extend the training set efficiently so that the coverage of the semantics in the training set wouldn't be obsolete.

show abstract

Section: Related Workmentioning

confidence: 99%

High-level semantic image annotation based on hot Internet topics

Wang

et al. 2013

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…A latent variable is introduced for capturing emergent patterns from the observations. This idea is inspired from topic models which was originally proposed for text [4,18] and had applications to other domains such as image retrieval and bioinformatics [20,41].…”

Section: The Probabilistic Modelmentioning

confidence: 99%

Human interaction discovery in smartphone proximity networks

Minh

Gática-Pérez

2011

Pers Ubiquit Comput

View full text Add to dashboard Cite

Since humans are fundamentally social beings and interact frequently with others in their daily life, understanding social context is of primary importance in building context-aware applications. In this paper, using smartphone Bluetooth as a proximity sensor to create social networks, we present a probabilistic approach to mine human interaction types in real life. Our analysis is conducted on Bluetooth data continuously sensed with smartphones for over one year from 40 individuals who are professionally or personally related. The results show that the model can automatically discover a variety of social contexts. We objectively validated our model by studying its predictive and retrieval performance.

show abstract

“…The idea of query expansion (QE) has been used in some previous research works in image retrieval [9,11,12,14]. In [14], the authors attempt to automatically annotate and retrieve images by applying QE in its relevance model based on a set of training images.…”

Section: Related Workmentioning

confidence: 99%

“…In [14], the authors attempt to automatically annotate and retrieve images by applying QE in its relevance model based on a set of training images. They represent an image as a set of blobs resulted from clustering on image features as well as a set of keywords.…”

Section: Related Workmentioning

confidence: 99%

Exploring statistical correlations for image retrieval

Wang

2006

Multimedia Systems

View full text Add to dashboard Cite

Bridging the cognitive gap in image retrieval has been an active research direction in recent years, of which a key challenge is to get enough training data to learn the mapping functions from low-level feature spaces to high-level semantics. In this paper, image regions are classified into two types: key regions representing the main semantic contents and environmental regions representing the contexts. We attempt to leverage the correlations between types of regions to improve the performance of image retrieval. A Context Expansion approach is explored to take advantages of such correlations by expanding the key regions of the queries using highly correlated environmental regions according to an image thesaurus. The thesaurus serves as both a mapping function between image low-level features and concepts and a store of the statistical correlations between different concepts. It is constructed through a data-driven approach which uses Web data (images, their surrounding textual annotations) as training data source to learn the region concepts and to explore the statistical correlations. Experimental results on a database of 10,000 general-purpose images show the effectiveness of our proposed approach in both improving search precision (i.e. filter irrelevant images) and recall (i.e. retrieval relevant images whose context may be varied). Several major factors which have impact on the performance of our approach are also studied.

show abstract

Automatic image annotation and retrieval using cross-media relevance models

Cited by 823 publications

References 19 publications

High-level semantic image annotation based on hot Internet topics

High-level semantic image annotation based on hot Internet topics

Human interaction discovery in smartphone proximity networks

Exploring statistical correlations for image retrieval

Contact Info

Product

Resources

About