Beyond active noun tagging: Modeling contextual interactions for multi-class active learning

Abstract

We present an active learning framework to simultaneously learn appearance and contextual models for scene understanding tasks (multi-class classification). Existing multi-class active learning approaches have focused on utilizing classification uncertainty of regions to select the most ambiguous region for labeling. These approaches, however, ignore the contextual interactions between different regions of the image and the fact that knowing the label for one region provides information about the labels of other regions. For example, the knowledge of a region being sea is informative about regions satisfying the "on" relationship with respect to it, since they are highly likely to be boats. We explicitly model the contextual interactions between regions and select the question which leads to the maximum reduction in the combined entropy of all the regions in the image (image entropy). We also introduce a new methodology of posing labeling questions, mimicking the way humans actively learn about their environment. In these questions, we utilize the regions linked to a concept with high confidence as anchors, to pose questions about the uncertain regions. For example, if we can recognize water in an image then we can use the region associated with water as an anchor to pose questions such as "what is above water?". Our active learning framework also introduces questions which help in actively learning contextual concepts. For example, our approach asks the annotator: "What is the relationship between boat and water?" and utilizes the answer to reduce the image entropies throughout the training dataset and obtain more relevant training examples for appearance models.

Active Learning for Human Pose Estimation

“…Methods that intelligently design the query space [39,32,30] also share the spirit of reducing annotation effort. Other works have looked into active learning schemes that query for multiple types of annotator feedback [50,4,43]. In this paper, we propose a new computer assisted annotation interface for human pose estimation.…”

Multi-level Adaptive Active Learning for Scene Classification

“…[30] treats the overall object classification problem as a multi-instance learning problem and considers the same type of labels at two levels, instance level (segments) and bag level (images). These works [18,30] nevertheless are still limited to exploiting the same type of standard queries, while another few works [1,21,27,11] have exploited semantic or multiple types of queries. [1,21] introduces a new interactive learning paradigm that allows the supervisor to additionally convey useful domain knowledge using relative attributes.…”

“…[1,21] introduces a new interactive learning paradigm that allows the supervisor to additionally convey useful domain knowledge using relative attributes. [27] presents an active learning framework to simultaneously learn appearance and contextual models for scene understanding. It explores three different types of questions: regional labeling questions, linguistic questions and contextual questions.…”

Attributes for Classifier Feedback

“…Our approach uses a human in the loop only during training. In [29], the learner actively asks the supervisor linguistic questions involving prepositions and attributes. In contrast, in our work, an informative attribute is selected by the supervisor that allows the classifier to better learn from its mistakes.…”