Probabilistic Matrix Factorization With Semantic And Visual Neighborhoods For Image Tag Completion

Abstract

We present an image tag completion method, namely PMF-SVN, where the key idea is to exploit images' Semantically and Visually similar Neighborhoods (SVNs) in the learning process of a Probabilistic Matrix Factorization (PMF) framework. We propose a two-step SVN formation algorithm that can generate an image set with the images being both visually and semantically similar. Furthermore, we introduce an efficient way to incorporate the formed SVNs into the learning process of PMF, under the constraint that the latent features of each image are averaged by the features of the images that belong to its SVN. In our experiments with benchmark datasets, we show that the proposed PMF-SVN method outperforms competitive baselines, in terms of completion accuracy, by efficiently capturing the semantical and visual associations between images and tags in SVNs.