A hierarchical recurrent approach to predict scene graphs from a visual‐attention‐oriented perspective

Abstract

A scene graph provides a powerful intermediate knowledge structure for various visual tasks, including semantic image retrieval, image captioning, and visual question answering. In this paper, the task of predicting a scene graph for an image is formulated as two connected problems, ie, recognizing the relationship triplets, structured as <subject‐predicate‐object>, and constructing the scene graph from the recognized relationship triplets. For relationship triplet recognition, we develop a novel hierarchical recurrent neural network with visual attention mechanism. This model is composed of two attention‐based recurrent neural networks in a hierarchical organization. The first network generates a topic vector for each relationship triplet, whereas the second network predicts each word in that relationship triplet given the topic vector. This approach successfully captures the compositional structure and contextual dependency of an image and the relationship triplets describing its scene. For scene graph construction, an entity localization approach to determine the graph structure is presented with the assistance of available attention information. Then, the procedures for automatically converting the generated relationship triplets into a scene graph are clarified through an algorithm. Extensive experimental results on two widely used data sets verify the feasibility of the proposed approach.

Development, Validation and Comparison of Artificial Neural Network Models and Logistic Regression Models Predicting Survival of Unresectable Pancreatic Cancer

“…Artificial neural networks have been developed as an effective statistical technique in the last 40 years (Dayhoff and DeLeo, 2001). They have been used in many fields and established as viable computational methodologies in computer science, biochemical and medical fields (Baxt and Skora, 1996;Milik et al, 1998;Gao et al, 2019;Yin et al, 2019;Deng et al, 2020;Yu et al, 2020). The network itself consists of an input layer, one or more hidden layers, and an output layer.…”

Automated Detection of Acute Lymphoblastic Leukemia From Microscopic Images Based on Human Visual Perception

“…In this work, we implemented sixteen widely used features, of which nine had morphological characteristics and seven had statistical characteristics ( Bodzas, 2019 , p. 51). Another approach to extract features is the use of a convolution neural network model, which extracts a collection of feature vectors ( Gao et al, 2019 ). In contrast to our approach, this feature space does not carry fully comprehensible information, and therefore cannot be interpreted in deep detail.…”

Incorporating Intelligence in Fish Feeding System for Dispensing Feed Based on Fish Feeding Intensity

“…In any type of developing pattern classification algorithms, evaluation of network performance is essential in order to access the system accuracy [41]- [43]. In the present study, the system performance was evaluated using k-fold crossvalidation [44], [45].…”