Mining social images with distance metric learning for automated image tagging

Wu, Pengcheng; Hoi, Steven C. H.; Zhao, Peilin; He, Ying

doi:10.1145/1935826.1935865

Cited by 51 publications

(32 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Van Zwol [Van Zwol and Garcia Pueyo 2012] proposed a spatially-aware image retrieval method. Wu [Wu et al 2011] investigated distance metric learning for image tagging. San Pedro [San Pedro et al 2012] leverages user comments for image re-ranking.…”

Section: Related Workmentioning

confidence: 99%

“…Traditional non-microblog works mainly focus on either image annotation or text illustration alone, and therefore cannot solve all associations between pictures and words in multimedia-rich microblogs. These works assume either that explicit association between visible parts of the picture with nouns and verbs [Wu et al 2011;Van Zwol and Garcia Pueyo 2012], or that a picture is an illustration of a document with many words [Joshi et al 2006]. However, disobeying the fundamental assumptions of traditional annotation, tagging, and retrieval systems, the pictures and words in multimedia-rich microblogs are loosely associated and a correspondence between pictures and words cannot be established .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs

Wang

Cui

Xie

et al. 2014

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Nowadays, the amount of multimedia contents in microblogs is growing significantly. More than 20% of microblogs link to a picture or video in certain large systems. The rich semantics in microblogs provide an opportunity to endow images with higherlevel semantics beyond object labels. However, this arises new challenges for understanding the association between multimodal multimedia contents in multimedia-rich microblogs. Disobeying the fundamental assumptions of traditional annotation, tagging, and retrieval systems, pictures and words in multimedia-rich microblogs are loosely associated and a correspondence between pictures and words cannot be established. To address the aforementioned challenges, we present the first study analyzing and modeling the associations between multimodal contents in microblog streams, aiming to discover multimodal topics from microblogs by establishing correspondences between pictures and words in microblogs. We first use a data-driven approach to analyze the new characteristics of the words, pictures and their association types in microblogs. We then propose a novel generative model, called the Bilateral Correspondence Latent Dirichlet Allocation (BC-LDA) model. Our BC-LDA model can assign flexible associations between pictures and words, and is able to not only allow picture-word co-occurrence with bilateral directions, but also single modal association. This flexible association can best fit the data distribution, so that the model can discover various types of joint topics and generate pictures and words with the topics accordingly. We evaluate this model extensively on a large-scale real multimedia-rich microblogs dataset. We demonstrate the advantages of the proposed model in several application scenarios, including image tagging, text illustration and topic discovery. The experimental results demonstrate that our proposed model can significantly and consistently outperform traditional approaches.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs

Wang

Cui

Xie

et al. 2014

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

show abstract

“…Monay et al [19] proposed to annotate the image in a latent semantic space. Wu and Hoi et al [46], [49], [33] proposed to learn a metric Fig. 1.…”

Section: Related Workmentioning

confidence: 99%

“…Besides the classification approaches, several advanced machine learning approaches have been applied to image annotation, including annotation by search [54], tag propagation [38], probabilistic relevant component analysis (pRCA) [33], distance metric learning [19], [46], [49], [28], Tag transfer [15], and reranking [61]. Similar to the classification based approaches for image annotation, to achieve good performance, these approaches require a large number of well annotated images, and therefore are not suitable for the tag completion problem.…”

Section: Introductionmentioning

confidence: 99%

Tag Completion for Image Retrieval

Jin

Jain

2013

IEEE Trans. Pattern Anal. Mach. Intell.

212

View full text Add to dashboard Cite

Abstract-Many social image search engines are based on keyword/tag matching. This is because tag based image retrieval (TBIR) is not only efficient but also effective. The performance of TBIR is highly dependent on the availability and quality of manual tags. Recent studies have shown that manual tags are often unreliable and inconsistent. In addition, since many users tend to choose general and ambiguous tags in order to minimize their efforts in choosing appropriate words, tags that are specific to the visual content of images tend to be missing or noisy, leading to a limited performance of TBIR. To address this challenge, we study the problem of tag completion where the goal is to automatically fill in the missing tags as well as correct noisy tags for given images. We represent the image-tag relation by a tag matrix, and search for the optimal tag matrix consistent with both the observed tags and the visual similarity. We propose a new algorithm for solving this optimization problem. Extensive empirical studies show that the proposed algorithm is significantly more effective than the state-of-the-art algorithms. Our studies also verify that the proposed algorithm is computationally efficient and scales well to large databases.

show abstract

“…The model-based approaches heavily rely on pre-trained classifiers with machine learning algorithms [17] [19] [27] [30], while the model-free approach propagates tags through the tagging behavior of visual neighbors [18] [29]. The two streams of approaches both assume that there is a well-labelled image database (source domain) that has the same or at least a similar data distribution as the target domain, so that the well-labelled database can ensure good generalization abilities for both classifier training and tag propagation.…”

Section: Introductionmentioning

confidence: 99%

Tagging Personal Photos with Transfer Deep Learning

Mei

Yang

et al. 2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

The advent of mobile devices and media cloud services has led to the unprecedented growing of personal photo collections. One of the fundamental problems in managing the increasing number of photos is automatic image tagging. Existing research has predominantly focused on tagging general Web images with a welllabelled image database, e.g., ImageNet. However, they can only achieve limited success on personal photos due to the domain gaps between personal photos and Web images. These gaps originate from the differences in semantic distribution and visual appearance. To deal with these challenges, in this paper, we present a novel transfer deep learning approach to tag personal photos. Specifically, to solve the semantic distribution gap, we have designed an ontology consisting of a hierarchical vocabulary tailored for personal photos. This ontology is mined from 10, 000 active users in Flickr with 20 million photos and 2.7 million unique tags. To deal with the visual appearance gap, we discover the intermediate image representations and ontology priors by deep learning with bottomup and top-down transfers across two domains, where Web images are the source domain and personal photos are the target. Moreover, we present two modes (single and batch-modes) in tagging and find that the batch-mode is highly effective to tag photo collections. We conducted personal photo tagging on 7,000 real personal photos and personal photo search on the MIT-Adobe FiveK photo dataset. The proposed tagging approach is able to achieve a performance gain of 12.8% and 4.5% in terms of NDCG@5, against the state-of-the-art hand-crafted feature-based and deep learning-based methods, respectively. Categories and Subject Descriptors

show abstract

Mining social images with distance metric learning for automated image tagging

Cited by 51 publications

References 24 publications

Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs

Bilateral Correspondence Model for Words-and-Pictures Association in Multimedia-Rich Microblogs

Tag Completion for Image Retrieval

Tagging Personal Photos with Transfer Deep Learning

Contact Info

Product

Resources

About