Modeling Semantic Aspects for Cross-Media Image Indexing

Abstract

To go beyond the query-by-example paradigm in image retrieval, there is a need for semantic indexing of large image collections for intuitive text-based image search. Different models have been proposed to learn the dependencies between the visual content of an image set and the associated text captions, then allowing for the automatic creation of semantic indices for unannotated images. The task, however, remains unsolved. In this paper, we present three alternatives to learn a Probabilistic Latent Semantic Analysis model (PLSA) for annotated images, and evaluate their respective performance for automatic image indexing. Under the PLSA assumptions, an image is modeled as a mixture of latent aspects that generates both image features and text captions, and we investigate three ways to learn the mixture of aspects. We also propose a more discriminative image representation than the traditional Blob histogram, concatenating quantized local color information and quantized local texture descriptors.The first learning procedure of a PLSA model for annotated images is a standard EM algorithm, which implicitly assumes that the visual and the textual modalities can be treated equivalently. The other two models are based on an asymmetric PLSA learning, allowing to constrain the definition of the latent space on the visual or on the textual modality. We demonstrate that the textual modality is more appropriate to learn a semantically meaningful latent space, which translates into improved annotation performance. A comparison of our learning algorithms with respect to recent methods on a standard dataset is presented, and a detailed evaluation of the performance shows the validity of our framework. Index TermsImage annotation, textual indexing, image retrieval, quantized local descriptors, latent aspect modeling.

High-level semantic image annotation based on hot Internet topics

“…Wu et al [33] proposed more accurate annotations by calculating the contribution of each word to its visibility model. Monay and Gatica-Perez [21] introduced a latent topic model, probabilistic Latent Semantic Analysis (pLSA), into an image annotation algorithm and trained two pLSA models based on the visual features and associated text, respectively. This method then merged the topics in both models by assigning weights according to the entropy of the visual word distribution.…”

Discovering Multipart Appearance Models from Captioned Images

“…A number of researchers have studied the problem of automatic image annotation in recent years [2][3][4][5][6]1]. Given cluttered images of multiple objects paired with noisy captions, these systems can learn meaningful correspondences between caption words and appearance models.…”

“…Most of these automatic annotation systems do not focus on learning such feature configurations. Often, appearance is modeled as a mixture of features (e.g., [5,3,6]) in which common part configurations are reflected in co-occurrence statistics but without spatial information. Similarly, the Markov random field model proposed by Carbonetto et al [4] can represent adjacency relationships but not spatial configurations.…”

Can Geotags Help Image Recognition?

“…As the discriminative method, we use Support Vector Machine (SVM). As the generative method, we use probabilistic latent topic mixture models [9].…”