On Modeling Context from Objects with a Long Short-Term Memory for Indoor Scene Recognition

Abstract

Automatic scene recognition is still regarded as an open challenge, even though there are reports of outperforming human accuracy. This is specially true for indoor scenes, since they can be well represented by their composing objects, which is highly variable information. Objects vary in angle, size, texture, besides being often partially occluded on crowded scenes. Even though Convolutional Neural Networks showed remarkable performance for most image-related problems, for indoor scenes the top performances were attributed to approaches that added object-level information to the methodology, modeling their intricate relationship. Knowing that Recurrent Neural Networks were designed to model structure from a given sequence of elements, only recently researchers started exploiting its advantages applied to the problem of scene recognition. Even though such works are usually below the state of the art performance, there is still plenty of room to unravel the full potential of recurrent methodologies. Thus, this work proposes representing an image as a sequence of object-level information, extracting highly semantic features from models pre-trained on an object-centric dataset, in order to feed a bidirectional Long Short-Term Memory network trained for scene classification. We perform a Many-to-Many training approach, such that each input outputs a corresponding scene prediction, allowing us to use each individual prediction to boost recognition with a weighted voting approach. To the best of our knowledge, our sequence representation, as well as our late fusion of predictions was little pursued by methods from the literature based on recurrent approaches for scene recognition. We evaluated our proposal on three widely known datasets for scene recognition: Scene15, MIT67 and SUN397, outperforming recurrent-based methods on MIT67, a dataset entirely dedicated to the problem of indoor scenes, while the others, which mix indoor and outdoor environments presented as a greater challenge for our approach. However, we were able to improve performance on all datasets over the most successful methods on the literature by pairing our work to a few of them in an ensemble of classifiers. Meaning a joint strategy with our method was beneficial for the task of scene classification.

Deep Learning for Scene Classification: A Survey

“…To take fully advantage of back-propagation, scene representations are extracted from end-to-end trainable CNNs, like DAG-CNN [37], MFAFVNet [31], VSAD [32], G-MS2F [39], and DL-CNN [22]. To focus on main content of the scene, object detection is used to capture salient regions, such as MetaObject-CNN [33], WELDON [34], SDO [35], and BiLSTM [36]. Since features from multiple CNN layers or multiple views are complementary, many literatures [19], [21], [76] and GIST [77].…”

“…[24], [40], [41] also explored their complementarity to improve performance. In addition, there exists many strategies (like attention mechanism, contextual modeling, multi-task learning with regularization terms) to enhance representation ability, such as CFA [24], BiLSTM [36], MAPNet [43], MSN [44], and LGN [45]. For datasets, because depth images from RGB-D cameras are not vulnerable to illumination changes, since 2015, researchers have started to explore RGB-D scene recognition.…”

“…Sequential Model. With the success of sequential models, such as RNN and LSTM, capturing the sequential information among local regions has shown promising performance for scene classification [36]. Spatial dependencies are captured from direct or indirect connections between each region and its surrounding neighbors.…”

“…In their framework, the outputs of all LSTM layers are concatenated to form a contextual representation. In addition, the works [36], [192] captured semantic contextual knowledge from variable salient regions. In [192], two types of representations, i.e., COOR and SOOR, are proposed to describe object-to-object relations.…”

BORM: Bayesian Object Relation Model for Indoor Scene Recognition

“…To utilize the object relation for scene recognition, spatial object-to-object relation is studied for RGB-D scene recognition [8]. Besides, a Long Short-Term Memory modeling method is proposed to investigate the object relation with ROI selection [9]. To utilize the object information in the scene, the object model [10] is proposed as complementary semantic information of the scene combined with ResNet to better interpret the given scene.…”