Using Semantic Features for Scene Classification: how Good do they Need to Be?

Abstract

Semantic scene classification is a useful, yet challenging problem in image understanding. Most existing systems are based on low-level features, such as color or texture, and succeed to some extent. Intuitively, semantic features, such as sky, water, or foliage, which can be detected automatically, should help close the so-called semantic gap and lead to higher scene classification accuracy. To answer the question of how accurate the detectors themselves need to be, we adopt a generally applicable scene classification scheme that combines semantic features and their spatial layout as encoded implicitly using a block-based method. Our scene classification results show that although our current detectors collectively are still inadequate to outperform low-level features under the same scheme, semantic features hold promise as simulated detectors can achieve superior classification accuracy once their own accuracies reach above a nontrivial 90%.

Scene Parsing Using Region-Based Generative Models

“…To obtain multiclass classification, we trained a SVM for each class to distinguish it from all others, and classified the image with the class whose SVM gave the maximum output. Further details are given in [3].…”

“…The most effective of these models uses pairwise spatial relationships between regions. 3) In Section V, we compare this model with three other generative models: an exact model that models the full joint distribution of the scene type and every semantic region in the image, one that models co-occurrence of these regions while ignoring the actual spatial relations, and one that treats these regions independently. 4) Finally, we compare our model with a discriminative model that uses high-level features and with one that uses low-level features.…”

Home Interior Classification using SIFT Keypoint Histograms

“…As in [2], we convert the image to LST space and split the image into blocks formed by an NxN grid. We then compute the mean and variance of each block's color band.…”

“…Spatial color moments are a state-of-the-art feature used to distinguish outdoor scenes [2][10]; we use them as a baseline feature for comparison, even though color is expected to be more salient for outdoor scenes than indoor ones. As in [2], we convert the image to LST space and split the image into blocks formed by an NxN grid.…”

“…Distinguishing between common rooms in indoor environments such as homes is a much more challenging problem. Features typically used for outdoor scenes in the literature, such as spatial color moments [2] [10], often fail in indoor environments because colors are a weak predictor of room type. Also, home interiors contain much extraneous data common to most or all room types that is useless for classification: homogeneous areas corresponding to blank walls, floors, and ceilings and generic "objects" such as doors or corners.…”