Topic Modeling of Multimodal Data: An Autoregressive Approach

Abstract

Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to deal with multimodal data, such as in image annotation tasks. Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for text document modeling. In this work, we show how to successfully apply and extend this model to multimodal data, such as simultaneous image classification and annotation. Specifically, we propose SupDocNADE, a supervised extension of DocNADE, that increases the discriminative power of the hidden topic features by incorporating label information into the training objective of the model and show how to employ SupDocNADE to learn a joint representation from image visual words, annotation words and class label information. We also describe how to leverage information about the spatial position of the visual words for SupDocNADE to achieve better performance in a simple, yet effective manner. We test our model on the LabelMe and UIUC-Sports datasets and show that it compares favorably to other topic models such as the supervised variant of LDA and a Spatial Matching Pyramid (SPM) approach.

A discriminative and sparse topic model for image classification and annotation

“…The proposed method is compared on image classification with the following five methods: sLDA-ann [32], abc-corr-LDA [24], SupDocNADE [38], SAGE [12] and MedSTC [37]. Although SAGE and MedSTC only consider image visual information, they add constraints to topic model for obtaining sparse topics, we compare their results for image classification.…”

“…Zheng et al [38] proposed the supervised document neural autoregressive distribution estimator (SupDocNADE) model to simultaneously deal with the image classification and annotation tasks. It obtains the hidden topic features by using the neural network on the mixed representation with all visual and annotation words, and learns the connection between hidden layer and the class label.…”

“…Recently, graph model [8,11], neural network model [38], supervised non-negative matrix factorization model [21], regression model [19], and generative graphical model [32,24] are adopted here. Among them, the graph model and the neural network model have higher complexities for the process of building graph or network.…”

Bayesian nonparametric inference of latent topic hierarchies for multimodal data

“…We then extracted visual words using a k-means algorithm, resulting in 1,000 visual word types. For the UIUC-Sports dataset, we used the same settings as used by Zheng et al [12]. We set the grid size to 8×8 pixels and set the scale to 16 pixels.…”

“…UIUC-Sports consists of 8 classes about sports. We used the dataset used by Zheng et al [12], where the number of images for each class ranges from 137 (bocce) to 330 (croquet), and the total number of images is 1,792.…”