Category-Based Deep CCA for Fine-Grained Venue Discovery From Multimodal Data

Abstract

In this work, travel destinations and business locations are taken as venues. Discovering a venue by a photograph is very important for visual context-aware applications. Unfortunately, few efforts paid attention to complicated real images such as venue photographs generated by users. Our goal is fine-grained venue discovery from heterogeneous social multimodal data. To this end, we propose a novel deep learning model, category-based deep canonical correlation analysis. Given a photograph as input, this model performs: 1) exact venue search (find the venue where the photograph was taken) and 2) group venue search (find relevant venues that have the same category as the photograph), by the cross-modal correlation between the input photograph and textual description of venues. In this model, data in different modalities are projected to a same space via deep networks. Pairwise correlation (between different modality data from the same venue) for exact venue search and category-based correlation (between different modality data from different venues with the same category) for group venue search are jointly optimized. Because a photograph cannot fully reflect rich text description of a venue, the number of photographs per venue in the training phase is increased to capture more aspects of a venue. We build a new venue-aware multimodal data set by integrating Wikipedia featured articles and Foursquare venue photographs. Experimental results on this data set confirm the feasibility of the proposed method. Moreover, the evaluation over another publicly available data set confirms that the proposed method outperforms state of the arts for cross-modal retrieval between image and text.

Weakly Supervised Instance Segmentation Based on Two-Stage Transfer Learning

“…This type of methods concentrate on mapping instances from the source domain and target domain into a new data space. Yu et al [66] aimed at fine-grained venue discovery from heterogeneous social multi-model data. Yu et al [67] focused on cross-modal correlation learning between audio modality and text modality.…”

SVDTWDD Method for High Correct Recognition Rate Classifier With Appropriate Rejection Recognition Regions

“…Over the past decades, many prominent results [1]- [29] in signal processing and in the design of classifiers have been achieved by geometric and algebra. However, at present, feature regions of the same class determined by Support Vector Machines (SVM) classifier, Support Vector Domain Description (SVDD) classifier and Deep Learning (DL) classifier may occupy feature regions of other classes or unknown classes.…”

“…There is a risk that samples of other classes or unknown classes are wrongly classified as a known class. Hence, some SVM classifiers with high correct recognition rate, SVDD classifiers and DL classifiers still have about 2% error recognition rate [6]- [29]. These classifiers cannot be directly applied to the serious authentication and recognition applications, such as major disease detection, human identity authentication, and identification of banknote, bill, or terrorist.…”

Learning Joint Embedding for Cross-Modal Retrieval

“…Cluster-CCA [8] establishes possible one-to-one correspondences to contain the category information during training stage. The category-based DCCA (C-DCCA) [9] not only considers the instance-based correlation but also learns the category-based correlation.…”