Object-Level Context Modeling For Scene Classification with Context-CNN

Abstract

Convolutional Neural Networks (CNNs) have been used extensively for computer vision tasks and produce rich feature representation for objects or parts of an image. But reasoning about scenes requires integration between the low-level feature representations and the high-level semantic information.We propose a deep network architecture which models the semantic context of scenes by capturing object-level information. We use Long Short Term Memory(LSTM) units in conjunction with object proposals to incorporate object-object relationship and object-scene relationship in an end-to-end trainable manner. We evaluate our model on the LSUN dataset and achieve results comparable to the state-of-art. We further show visualization of the learned features and analyze the model with experiments to verify our model's ability to model context.

On Modeling Context from Objects with a Long Short-Term Memory for Indoor Scene Recognition

“…It is also worth mentioning the work of Javed and Nelakanti [2017], a recent proposal for scene recognition that also assumes object features as an ideal source of information to recognize scenes, reinforcing our premise. This work showed that with a small fraction of a large-scale dataset, one can be competitive with the state of the Figure 3.6: Hierarchical recurrent approach proposed by Zuo et al [2016].…”

“…. 3.7 Methodology proposed by Javed and Nelakanti [2017]. A combination of CNN and RNN architectures to model spatial context for scene recognition.…”

“…This is relevant because it means the output sequence based on a region proposal approach has variable length. To the best of our knowledge, there are only a couple of works on the topic of scene recognition which exploits such an approach to represent a single image as a sequence, and they choose to fix the sequence length despite the aforementioned behavior of region proposal methods [Javed and Nelakanti, 2017;Wang and Pan, 2017].…”

“…There is also a free parameter on the recurrent layer concerning its hidden size, i.e., the size of its output h t (refer to Equation 2.3). Considering that we tested different architectures and training approaches, as it will later be showed on Section 5.7, h t was fixed to 512 as proposed by the work of Javed and Nelakanti [2017]. It is worth reminding that our recurrent layer is bidirectional, which means that although h t = 512, the actual outputs is 2 × h t since the output of both recurrent units (one for each direction) will be concatenated before feeding the next layer.…”

Deep Learning for Scene Classification: A Survey

“…Herein, COOR adapts the cooccurring frequency to represent object-to-object relations, while SOOR is encoded with sequential model via regarding object sequences as sentences. Rooted in the work of Javed et al [193], Laranjeira et al [36] proposed a bidirectional LSTM to capture the contextual relations of regions of interest, as shown in Fig. 11 (a).…”