Multilayer pLSA for multimodal image retrieval

Abstract

It is current state of knowledge that our neocortex consists of six layers [10]. We take this knowledge from neuroscience as an inspiration to extend the standard single-layer probabilistic Latent Semantic Analysis (pLSA) [13] to multiple layers. As multiple layers should naturally handle multiple modalities and a hierarchy of abstractions, we denote this new approach multilayer multimodal probabilistic Latent Semantic Analysis (mm-pLSA). We derive the training and inference rules for the smallest possible non-degenerated mm-pLSA model: a model with two leaf-pLSAs (here from two different data modalities: image tags and visual image features) and a single top-level pLSA node merging the two leaf-pLSAs. From this derivation it is obvious how to extend the learning and inference rules to more modalities and more layers. We also propose a fast and strictly stepwise forward procedure to initialize bottom-up the mm-pLSA model, which in turn can then be post-optimized by the general mm-pLSA learning algorithm. We evaluate the proposed approach experimentally in a query-by-example retrieval task using 50-dimensional topic vectors as image models. We compare various variants of our mm-pLSA system to systems relying solely on visual features or tag features and analyze possible pitfalls of the mm-pLSA training. It is shown that the best variant of the the proposed mm-pLSA system outperforms the unimodal systems by approximately 19% in our query-by-example task.

Tripartie Graph Models for Multi Modal Retrieval

“…This is similar to the methods in [15]. We also compare our multimodal pLSA with multilayer multi modal pLSA(mm-pLSA) proposed by [11]. From the comparative study on a variety of popular data sets, we find that the results obtained by direct method are superior to other methods.…”

“…Then a maximal margin classifier is applied for retrieval. Romberg et al [11] proposed a mmpLSA, with two separate leaf-pLSAs, and a single top level pLSA node merging the two leaf-pLSAs. Here, they apply pLSA to each mode, i.e., visual features and textual words separately, and then concatenate the derived topic vectors of each mode to learn another pLSA on top of that.…”

“…There are also attempts for extending them to multimodal data [3,11,14]. However, all of them require complex mathematical computations involving large matrices.…”

Toward a higher-level visual representation for content-based image retrieval

“…This method aims at eliminating the most noisy words generated by the vocabulary building process, using multilayer pLSA. Lienhart et al [18] proposed a multilayer multi-modal probabilistic Latent Semantic Analysis (mm-pLSA). The proposed approach (mm-pLSA) has two modes: one mode for visual words and the other one for image tags.…”

Unified hypergraph for image ranking in a multimodal context

“…Once the graph is constructed, standard graph mining techniques such as Pagerank [4] can be used to identify the "authority" vertices (images). Besides image visual features, it has been shown in previous work [5] that integrating other modalities can boost retrieval performance. In the work of [6], graphs have been used to convey multimodal information for search and retrieval.…”