Image Parsing: Unifying Segmentation, Detection, and Recognition

Abstract

In this paper we present a Bayesian framework for parsing images into their constituent visual patterns. The parsing algorithm optimizes the posterior probability and outputs a scene representation in a "parsing graph", in a spirit similar to parsing sentences in speech and natural language. The algorithm constructs the parsing graph and re-configures it dynamically using a set of reversible Markov chain jumps. This computational framework integrates two popular inference approaches -generative (top-down) methods and discriminative (bottom-up) methods. The former formulates the posterior probability in terms of generative models for images defined by likelihood functions and priors. The latter computes discriminative probabilities based on a sequence (cascade) of bottom-up tests/filters. In our Markov chain design, the posterior probability, defined by the generative models, is the invariant (target) probability for the Markov chain, and the discriminative probabilities are used to construct proposal probabilities to drive the Markov chain. Intuitively, the bottom-up discriminative probabilities activate top-down generative models. In this paper, we focus on two types of visual patterns -generic visual patterns, such as texture and shading, and object patterns including human faces and text. These types of patterns compete and cooperate to explain the image and so image parsing unifies image segmentation, object detection, and recognition (if we use generic visual patterns only then image parsing will correspond to image segmentation [46].). We illustrate our algorithm on natural images of complex city scenes and show examples where image segmentation can be improved by allowing object specific knowledge to disambiguate low-level segmentation cues, and conversely object detection can be improved by using generic visual patterns to explain away shadows and occlusions.

Image interpretation by a single bottom-up top-down cycle

“…Compared with past computational modeling, previous models have used part-based object recognition (31)(32)(33) and combined BU with TD processing (18,19,(34)(35)(36). However, past models did not study or report results on part recognition, did not examine the limitations of feed-forward models for part recognition, and did not demonstrate the contribution of a fast TD process to part detection and localization.…”

What Is a Good Image Segment? A Unified Approach to Segment Extraction

“…The large diversity of image segment types has increased the urge to devise a unified segmentation approach. Tu et al [13] provided such a unified probabilistic framework, which enables to "plug-in" a wide variety of parametric models capturing different segment types. While their framework elegantly unifies these parametric models, it is restricted to a predefined set of segment types, and each specific object/segment type (e.g., faces, text, texture etc.)…”

“…Our work builds on top of [14], providing a general segment quality score and a corresponding image segmentation algorithm, which applies to a large diversity of segment types, and can be applied for various segmentation tasks. Although general, our unified segmentation framework does not require any pre-definition or modelling of segment types (in contrast to the unified framework of [13]). …”

Learning to Combine Bottom-Up and Top-Down Segmentation

“…As can be seen, the top-down segmentation is better than any of the bottom-up segmentations but still misses important details. In recent years, several authors have therefore suggested combining top-down and bottom-up segmentation [2,21,17,6]. Borenstein et al [2] choose among a discrete set of possible low-level segmentations by minimizing a cost function that includes a bias towards the top-down segmentation.…”

“…Borenstein et al [2] choose among a discrete set of possible low-level segmentations by minimizing a cost function that includes a bias towards the top-down segmentation. In the image parsing framework of Tu et al [17] object-specific detectors serve as a proposal distribution for a data-driven Monte-Carlo sampling over possible segmentations. In the OBJ-CUT algorithm [6] a layered pictorial structure is used to define a bias term for a graph-cuts energy minimization algorithm (the energy favors segmentation boundaries occurring at image discontinuities).…”