Learning Structured Visual Detectors From User Input at Multiple Levels

Abstract

In this paper, we propose a new framework for the dynamic construction of structured visual object/scene detectors for content-based retrieval. In the Visual Apprentice, a user defines visual object/scene models via a multiple-level Definition Hierarchy: a scene consists of objects, which consist of object-parts, which consist of perceptual-areas, which consist of regions. The user trains the system by providing example images/videos and labeling components according to the hierarchy she defines (e.g., image of two people shaking hands contains two faces and a handshake). As the user trains the system, visual features (e.g., color, texture, motion, etc.) are extracted from each example provided, for each node of the hierarchy (defined by the user). Various machine learning algorithms are then applied to the training data, at each node, to learn classifiers. The best classifiers and features are then automatically selected for each node (using cross-validation on the training data). The process yields a Visual Object/Scene Detector (e.g., for a handshake), which consists of an hierarchy of classifiers as it was defined by the user. The Visual Detector classifies new images/videos by first automatically segmenting them, and applying the classifiers according to the hierarchy: regions are classified first, followed by the classification of perceptual-areas, object-parts and objects. We discuss how the concept of Recurrent Visual Semantics can be used to identify domains in which learning techniques such as the one presented can be applied. We then present experimental results using several hierarchies for classifying images and video shots (e.g., Baseball video, images that contain handshakes, skies, etc.). These results, which show good performance, demonstrate the feasibility and usefulness of dynamic approaches for constructing structured visual object/scene detectors from user input at multiple levels.

Incorporating feature hierarchy and boosting to achieve more effective classifier training and concept-oriented video summarization and skimming

“…One advantage of such rule-based approach is the ease to insert, delete, and modify the existing rules when the nature of video classes changes. Since the underlying video making rules are used for detecting the semantic video events, the rule-based approach is only attractive for some specific domains such as news video and film which have well-defined story structure for the semantic video events [Adams et al 2003;Naphade and Huang 2001;Jaimes and Chang 2001;Greenspan et al 2004;Qi et al 2003;Gatica-Perez et al 2003;Ma et al 2002;Smith and Kanade 1995;Arman et al 1994;He et al 1999;Sundaram et al 2002;Chang 2002;Sundaram and Chang 2002b]. On the other hand, surgery education videos do not have well-defined video making rules that can be used to generate the semantic video events, thus it is very important to develop new techniques that are able to achieve more effective video classification and concept-oriented video summarization and skimming.…”

“…He et al [1999] have also incorporated audio-visual information to summarize the audio-video presentations. To integrate the high-level video semantics for video summarization and skimming, hidden Markov models and Gaussian mixture models have been investigated widely for detecting semantic video scenes [Liu et al 1998;Ekin et al 2003;Xie et al 2003;Dimitrova et al 2000;Adams et al 2003;Naphade and Huang 2001;Jaimes and Chang 2001;Greenspan et al 2004;Qi et al 2003;Gatica-Perez et al 2003]. Naphade and Huang [2001] and Adams et al [2003] have also investigated the techniques for hierarchical video classification.…”

“…Bridging the semantic gap in general is very hard if not impossible, thus narrowing the video domain plays an important role on achieving more effective video classification [Pfeiffer et al 1999;Kender and Yeo 1998;Sundaram and Chang 2002a;Adames et al 2002;Haering et al 2000;Zhou et al 2000;Fischer et al 1995;Hanjalic et al 1999;Liu et al 1998;Ekin et al 2003;Xie et al 2003;Dimitrova et al 2000;Adams et al 2003;Naphade and Huang 2001;Jaimes and Chang 2001;Greenspan et al 2004;Qi et al 2003;Gatica-Perez et al 2003]. Based on this observation, our research focuses on one specific domain of surgery education videos, where the contextual relationships between the semantic video concepts and the relevant video patterns and their low-level perceptual features are better defined.…”

Image classification using multimedia knowledge networks

“…Many methods rely uniquely on perceptual features such as color histogram [4][9] [11]; whereas few also consider text features from annotations or captions [7] [8]. There are some approaches that only use individual classifiers or joint distributions [1]; while others combine multiple classifiers for improved accuracy [4][9] [11].…”

“…Finally, experts handpick the classes in many methods [11] [7][8] [9] to which the classifiers are often fine-tuned. Exceptions are frameworks where "expert" users define their own classes and relations [4], and approaches that associate words annotating images to new images or regions [1]. The most similar prior work is [7] and [9], which learn Bayesian Networks (BNs) with classifiers as nodes.…”