Multimedia search and retrieval using multimodal annotation propagation and indexing techniques

Abstract

In this paper, a novel framework for multimodal search and retrieval of rich media objects is presented. The searchable items are media representations consisting of multiple modalities, such as 2D images, 3D objects and audio files, which share a common semantic concept. A manifold learning technique based on Laplacian Eigenmaps was appropriately modified in order to merge the low-level descriptors of each separate modality and create a new low-dimensional multimodal feature space, where all media objects can be mapped irrespective of their constituting modalities. To accelerate search and retrieval and make the framework suitable even for web-scale applications, a multimedia indexing scheme is adopted, which represents each object of the dataset by the ordering of a number of reference objects. Moreover, the hubness property is introduced in this paper as a criterion to select the most representative reference objects, thus, achieving the maximum possible performance of indexing. The content-based similarity of the multimodal descriptors is also used to automatically annotate the objects of the dataset using a predefined set of attributes. Annotation propagation is utilized to approximate the multimodal descriptors for multimodal queries that do not belong to the dataset.

Topic correlation model for cross-modal multimedia information retrieval

“…One class of models are based on combination strategies: (1) combining low-level features from different modalities into concise multi-modal features. In [12], a manifold learning algorithm based on Laplacian Eigenmaps is introduced to combine low-level descriptors of each separate modality and map them to a new low-dimensional multi-modal feature space. In this feature space, semantically similar multi-modal data are represented by multi-modal descriptor vectors close to each other.…”

Augmented reality system using lidar point cloud data for displaying dimensional information of objects on mobile phones

“…Hierarchical image partitioning method is used to detect scale of a live image by matching this image with the database images. Lazaridis et al (2013) designed a method to search and retrieve multimodal data. This framework links images to semantic annotation using some similarity measure.…”

Search and retrieval of multi-modal data associated with image-parts

“…The main difficulty in cross-media retrieval is to define a similarity measure among heterogeneous low-level features. In order to simultaneously search and retrieve data from multiple modalities, other approaches have been considered [10,11,12,13,14,15,16,17,18,19,20]. For instance in [16], it is experimentally shown that multimodal queries achieve higher retrieval accuracy than mono-modal ones.…”

“…The work in [10] suggests using a combination of ontology browsing and keyword-based querying. The methods presented in [11,14,15,16] use a similar approach and rely on the assumption that every document has an equal number of nearest neighbors for each of the modalities. However, such an assumption might degrade the retrieval performance as a document containing "image+text" may have many nearest neighbors in image modality, but not as many relevant textual data.…”

“…Our approach utilizes the co-occurrence of data points within a modality and across multiple modalities using a graph structure. We note that the proposed work does not impose relevance between each document and an equal number of media objects from all modalities as assumed by [11,14,15,16]. In summary, our contributions are threefold:…”