Image Understanding using vision and reasoning through Scene Description Graph

Aditya, Somak; Yang, Yezhou; Baral, Chitta; Aloimonos, Yiannis; Fermüller, Cornelia

doi:10.1016/j.cviu.2017.12.004

Cited by 69 publications

(35 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among these works, [55] uses scene graphs for explainable and explicit reasoning with structured knowledge. Aditya et al [56] use directed and labeled scene description graph for reasoning in image captioning, retrieval, and visual question answering applications. In another recent work, [57] introduces a method for globally reasoning over regional relations in a single image.…”

Section: Related Workmentioning

confidence: 99%

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Liu

Adeli

Cao

et al. 2020

IEEE Robot. Autom. Lett.

143

View full text Add to dashboard Cite

Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on top of segmented object instances within and across video frames. Pedestrian intent, defined as the future action of crossing or not-crossing the street, is very crucial piece of information for autonomous vehicles to navigate safely and more smoothly. We approach the problem of intent prediction from two different perspectives and anticipate the intention-to-cross within both pedestrian-centric and location-centric scenarios. In addition, we introduce a new dataset designed specifically for autonomousdriving scenarios in areas with dense pedestrian populations: the Stanford-TRI Intent Prediction (STIP) dataset. Our experiments on STIP and another benchmark dataset show that our graph modeling framework is able to predict the intention-to-cross of the pedestrians with an accuracy of 79.10% on STIP and 79.28% on Joint Attention for Autonomous Driving (JAAD) dataset up to one second earlier than when the actual crossing happens. These results outperform baseline and previous work. Please refer to http://stip.stanford.edu/ for the dataset and code.Index Terms-spatiotemporal graphs, forecasting, graph neural networks, autonomous-driving. Recent work [19]-[23] introduced pedestrian intent prediction and have typically tackled the problem by observing pedestrian-specific features such as location, velocity, and

show abstract

Section: Related Workmentioning

confidence: 99%

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Liu

Adeli

Cao

et al. 2020

IEEE Robot. Autom. Lett.

143

View full text Add to dashboard Cite

show abstract

“…A stronger correlation with human judgements indicates that a metric captures the features that humans look for, while assessing a candidate caption. In order to measure the sentence-level correlation of our proposed metric with human judgements we use the COMPOS-ITE dataset [2] which contains human judgements for 11,985 candidate captions and their image counterparts. The images in this dataset are obtained from MS COCO, Flickr8k and Flickr30k datasets, whereas, the associated captions consist of human generated captions (sourced from the aforementioned datasets) and machine generated captions (using two captioning models [2], [19]).…”

Section: Correlation With Human Judgementsmentioning

confidence: 99%

NNEval: Neural Network Based Evaluation Metric for Image Captioning

Sharif

White

Bennamoun

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

The automatic evaluation of image descriptions is an intricate task, and it is highly important in the development and fine-grained analysis of captioning systems. Existing metrics to automatically evaluate image captioning systems fail to achieve a satisfactory level of correlation with human judgements at the sentence level. Moreover, these metrics, unlike humans, tend to focus on specific aspects of quality, such as the n-gram overlap or the semantic meaning. In this paper, we present the first learning-based metric to evaluate image captions. Our proposed framework enables us to incorporate both lexical and semantic information into a single learned metric. This results in an evaluator that takes into account various linguistic features to assess the caption quality. The experiments we performed to assess the proposed metric, show improvements upon the state of the art in terms of correlation with human judgements and demonstrate its superior robustness to distractions.

show abstract

“…Then, the region features appropriate for each layer are determined through two fully connected layers. and region features are extracted as follows: In the case of the relationship detection layer, the region features are determined by inputting the two bounding boxes ( , ) of the two objects comprising the relationship region, as well as the visual features, to the fully connected layer, as shown in (11). This aids identification of the spatial relationship (e.g., a relationship indicated by an "in" or "on" preposition) through the locational relationships between the two objects.…”

Section: Candidate Region Proposal As Shown Inmentioning

confidence: 99%

“…Further, the scene graph approach has lower difficulty as a learning problem, because there is no need to consider complex grammatical structures. Moreover, the knowledge graphs acquired from images can be easily combined with numerous existing background datasets and prior knowledge datasets and can potentially exert power in more application areas [11][12][13]. Image captions and scene graphs have common characteristics in that they are generated with consideration of the objects in the images and the relationships between those objects.…”

Section: Introductionmentioning

confidence: 99%

Deep Image Understanding Using Multilayered Contexts

Shin

Kim

2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Generation of scene graphs and natural language captions from images for deep image understanding is an ongoing research problem. Scene graphs and natural language captions have a common characteristic in that they are generated by considering the objects in the images and the relationships between the objects. This study proposes a deep neural network model named the Context-based Captioning and Scene Graph Generation Network (C2SGNet), which simultaneously generates scene graphs and natural language captions from images. The proposed model generates results through communication of context information between these two tasks. For effective communication of context information, the two tasks are structured into three layers: the object detection, relationship detection, and caption generation layers. Each layer receives related context information from the lower layer. In this study, the proposed model was experimentally assessed using the Visual Genome benchmark data set. The performance improvement effect of the context information was verified through various experiments. Further, the high performance of the proposed model was confirmed through performance comparison with existing models.

show abstract

Image Understanding using vision and reasoning through Scene Description Graph

Cited by 69 publications

References 16 publications

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

NNEval: Neural Network Based Evaluation Metric for Image Captioning

Deep Image Understanding Using Multilayered Contexts

Contact Info

Product

Resources

About