Interactively Guiding Semi-Supervised Clustering via Attribute-Based Explanations

Abstract

(ABSTRACT)Unsupervised image clustering is a challenging and often ill-posed problem. Existing image descriptors fail to capture the clustering criterion well, and more importantly, the criterion itself may depend on (unknown) user preferences. Semi-supervised approaches such as distance metric learning and constrained clustering thus leverage user-provided annotations indicating which pairs of images belong to the same cluster (must-link) and which ones do not (cannot-link). These approaches require many such constraints before achieving good clustering performance because each constraint only provides weak cues about the desired clustering.In this work, we propose to use image attributes as a modality for the user to provide more informative cues. In particular, the clustering algorithm iteratively and actively queries a user with an image pair. Instead of the user simply providing a must-link/cannot-link constraint for the pair, the user also provides an attribute-based reasoning e.g. "these two images are similar because both are natural and have still water" or "these two people are dissimilar because one is way older than the other". Under the guidance of this explanation, and equipped with attribute predictors, many additional constraints are automatically generated.We demonstrate the effectiveness of our approach by incorporating the proposed attributebased explanations in three standard semi-supervised clustering algorithms: Constrained K-Means, MPCK-Means, and Spectral Clustering, on three domains: scenes, shoes, and faces, using both binary and relative attributes.

Best of both worlds: Human-machine collaboration for object annotation

“…Recognition with humans in the loop. Among the most similar works to ours is the approaches which combine computer vision with human-in-the-loop collaboration for tasks such as fine-grained image classification [6,59,12,60], image segmentation [26], attribute-based classification [32,40,3], image clustering [34], image annotation [54,55,47], and human interaction [31] and object annotation in videos [58]. Methods such as [6,59,12,60] jointly model human and computer uncertainty and characterize human time versus annotation accuracy, but only incorporate a single type of human response.…”

“…The field of crowd engineering has provided lots of insight into human-machine collaboration for solving difficult problems in computing such as protein folding [41,9], disaster relief distribution [18] and galaxy discovery [38]. In computer vision with human-in-the-loop approaches, human intervention has ranged from binary question-and-answer [6,59,60] to attribute-based feedback [40,39,34] to free-form object annotation [58]. For understanding all objects in an image, one important decision is which questions to pose to humans.…”

Crowdsourcing in Computer Vision

“…Attribute-based feedback has also been used for interactive clustering, where the goal is not to name the object present in the image but rather to cluster a large collection of images in a meaningful way [Lad and Parikh, 2014].…”

Knowing who to listen to: Prioritizing experts from a diverse ensemble for attribute personalization

“…Attributes have been used extensively for a variety of applications in computer vision such as object recognition [20,21], scene understanding [22,23], image description [4,24], image search [5], clustering [25] and fine-grained recognition [26]. Many existing methods learn the attributes independently [23, 2, 1], often using linear models.…”