Scene Graph with 3D Information for Change Captioning

Liao, Zeming; Huang, Qingbao; Liang, Yu; Fu, Mingyi; Cai, Yi; Li, Qing

doi:10.1145/3474085.3475712

Cited by 15 publications

(2 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such volumetric methods have been extended to utilize semantics for more complete change detection [41], and even during online operation [5]. Similar to our work, Liao et al [42] demonstrate how semantic scene graphs can improve change captioning in simple 3D scenes.…”

Section: B Changes In Scene Semanticssupporting

confidence: 75%

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

Looper¹,

Rodríguez-Puigvert²,

Siegwart³

et al. 2022

Preprint

View full text Add to dashboard Cite

Numerous applications require robots to operate in environments shared with other agents such as humans or other robots. However, such shared scenes are typically subject to different kinds of long-term semantic scene changes. The ability to model and predict such changes is thus crucial for robot autonomy. In this work, we formalize the task of semantic scene variability estimation and identify three main varieties of semantic scene change: changes in the position of an object, its semantic state, or the composition of a scene as a whole.To represent this variability, we propose the Variable Scene Graph (VSG), which augments existing 3D Scene Graph (SG) representations with the variability attribute, representing the likelihood of discrete long-term change events. We present a novel method, DeltaVSG, to estimate the variability of VSGs in a supervised fashion. We evaluate our method on the 3RScan long-term dataset, showing notable improvements in this novel task over existing approaches. Our method DeltaVSG achieves a precision of 72.2% and recall of 66.8%, often mimicking human intuition about how indoor scenes change over time. We further show the utility of VSG predictions in the task of active robotic change detection, speeding up task completion by 62.4% compared to a scene-change-unaware planner. We make our code available as open-source.

show abstract

Section: B Changes In Scene Semanticssupporting

confidence: 75%

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

Looper¹,

Rodríguez-Puigvert²,

Siegwart³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The R-SCAN model by Lee et al [62] stands out for its focus on learning visual relationship features and proposing pre-training SGG models with relevant visual relationship data. Liao et al [67] ventured into the realm of 3D Scene Graph-based Change Captioning (SGCC), aiming to boost object location accuracy in change captioning tasks. Wu et al [118] introduced a method that integrates high-level visual concepts and external knowledge into a deep learning cascade of CNN and RNN, resulting in significant improvements in image captioning and VQA performance.…”

Section: Image Captioningmentioning

confidence: 99%

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

Khan,

Ilievski,

Breslin

et al. 2024

NAI

View full text Add to dashboard Cite

Combining deep learning and common sense knowledge via neurosymbolic integration is essential for semantically rich scene representation and intuitive visual reasoning. This survey paper delves into data- and knowledge-driven scene representation and visual reasoning approaches based on deep learning, common sense knowledge and neurosymbolic integration. It explores how scene graph generation, a process that detects and analyses objects, visual relationships and attributes in scenes, serves as a symbolic scene representation. This representation forms the basis for higher-level visual reasoning tasks such as visual question answering, image captioning, image retrieval, image generation, and multimodal event processing. Infusing common sense knowledge, particularly through the use of heterogeneous knowledge graphs, improves the accuracy, expressiveness and reasoning ability of the representation and allows for intuitive downstream reasoning. Neurosymbolic integration in these approaches ranges from loose to tight coupling of neural and symbolic components. The paper reviews and categorises the state-of-the-art knowledge-based neurosymbolic approaches for scene representation based on the types of deep learning architecture, common sense knowledge source and neurosymbolic integration used. The paper also discusses the visual reasoning tasks, datasets, evaluation metrics, key challenges and future directions, providing a comprehensive review of this research area and motivating further research into knowledge-enhanced and data-driven neurosymbolic scene representation and visual reasoning.

show abstract

Distractors-Immune Representation Learning with Cross-Modal Contrastive Regularization for Change Captioning

Tu,

Li,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Scene Graph with 3D Information for Change Captioning

Cited by 15 publications

References 27 publications

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

Distractors-Immune Representation Learning with Cross-Modal Contrastive Regularization for Change Captioning

Contact Info

Product

Resources

About