Learning to Tag from Open Vocabulary Labels

Abstract

Abstract. Most approaches to classifying media content assume a fixed, closed vocabulary of labels. In contrast, we advocate machine learning approaches which take advantage of the millions of free-form tags obtainable via online crowd-sourcing platforms and social tagging websites. The use of such open vocabularies presents learning challenges due to typographical errors, synonymy, and a potentially unbounded set of tag labels. In this work, we present a new approach that organizes these noisy tags into well-behaved semantic classes using topic modeling, and learn to predict tags accurately using a mixture of topic classes. This method can utilize an arbitrary open vocabulary of tags, reduces training time by 94% compared to learning from these tags directly, and achieves comparable performance for classification and superior performance for retrieval. We also demonstrate that on open vocabulary tasks, human evaluations are essential for measuring the true performance of tag classifiers, which traditional evaluation methods will consistently underestimate. We focus on the domain of tagging music clips, and demonstrate our results using data collected with a human computation game called TagATune.

Learning contextualized semantics from co-occurring terms via a Siamese architecture

“…Obviously, for a document, its one-hot representation out of the document list is unable to measure similarity between documents and is also subject to the generalization limitation. In recent work of Law, Settles & Mitchell (2010), Latent Direchlet Allocation (LDA) was used to represent terms in form of topics collectively and then a model was trained to map the acoustic content onto the topical representation to facilitate MMIR. Motivated by their work, we employ LDA to represent the local context in our work due to the generality of LDA in representing patterns of collective use.…”

“…Such semantics provides direct conceptlevel knowledge regarding the concerned multimedia objects. Typical applications include music crowd tagging services (Law, Settles & Mitchell, 2010) and multi-object image data set analysis (Rabinovich et al, 2007). Thanks to crowd-sourced annotation (Turnbull, Barrington & Lanckriet, 2008) and game-based tags collection (Law et al, 2007), large collections of descriptive terms are now accessible.…”

Multi-Tasking with Joint Semantic Spaces for Large-Scale Music Annotation and Retrieval

“…Finally, some machine learning algorithm is trained over these features in order to obtain a classifier for each tag. Often, the machine learning algorithm attempts to model the semantic relations between the tags [11]. A few state-of-the-art automatic annotation systems are briefly described in section 6.3.…”

Exploiting tag and word correlations for improved webpage clustering

“…A problem with such an approach is the large number of tags which leads to scalability issues. Furthermore, tags can potentially come from an open-vocabulary and be sparse [28]. Another issue could be synonymy where two different tags may have the same meaning.…”

“…Another issue could be synonymy where two different tags may have the same meaning. To address these issues in the context of music clip tag prediction, [28] proposed a framework that organizes tags into semantically meaningful classes using topic models, and then predicts these classes given a non-tagged piece of music. Such an approach can be useful for webpage tag prediction as well.…”