Recovering Surface Layout from an Image

Abstract

Humans have an amazing ability to instantly grasp the overall 3D structure of a scene-ground orientation, relative positions of major landmarks, etc.-even from a single image. This ability is completely missing in most popular recognition algorithms, which pretend that the world is flat and/or view it through a patch-sized peephole. Yet it seems very likely that having a grasp of this "surface layout" of a scene should be of great assistance for many tasks, including recognition, navigation, and novel view synthesis.In this paper, we take the first step towards constructing the surface layout, a labeling of the image into geometric classes. Our main insight is to learn appearance-based models of these geometric classes, which coarsely describe the 3D scene orientation of each image region. Our multiple segmentation framework provides robust spatial support, allowing a wide variety of cues (e.g., color, texture, and perspective) to contribute to the confidence in each geometric label. In experiments on a large set of outdoor images, we evaluate the impact of the individual cues and design choices in our algorithm. We further demonstrate the applicability of our method to indoor images, describe potential applications, and discuss extensions to a more complete notion of surface layout.

Superpixel Analysis for Object Detection and Tracking with Application to UAV Imagery

“…Practically none of the ground pixels are mislabeled as sky. These numbers agree quite well with the confusion matrix in [24].…”

“…It is thus desirable to differentiate the two based on region characteristics such as color. There are many options here, but we chose to use the surface layout classifier of [8,24]. This method divides images into geometric classes corresponding to three kinds of surfaces: support (aka ground), vertical, or sky.…”

“…Feeding these to the author-provided implementation of the scene layout algorithm (the later version from [24]) with the outdoor-trained version of their classifier yields the classifications shown in the third row of the figure. The scene layout labels are indicated by tinting: blue for sky, red for vertical, and green for support.…”

Detecting Changes in Images of Street Scenes

“…We choose the superpixels obtained by color based over segmentation scheme proposed in [24]. The choice of features has been adopted from [25] where each superpixel is characterized by location and shape (position of the centroid, relative position, number of pixels and area in the image), color (color histograms of RGB, HSV values and saturation value), texture (mean absolute response of the filter bank of 15 filters and histogram of maximum responses) and perspective cues computed from long linear segments and lines aligned with different vanishing points. The entire feature vector is of 194 dimensions.…”

Object Class Segmentation Using Reliable Regions

“…A more commonly used strategy is to integrate the results over multiple di erent segmentations of the same scene [6,18,13,19]. Usually several unsupervised bottom-up segmentation algorithms, each with di erent parameters are used.…”