Improving Visual Relation Detection using Depth Maps

Sharifzadeh, Sahand; Baharlou, Sina Moayed; Berrendorf, Max; Koner, Rajat; Tresp, Volker

doi:10.48550/arxiv.1905.00966

Cited by 5 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…showed how a prior distribution derived from triple occurrences could significantly improve on pure vision-based approaches and on approaches that used prior distributions derived from language models. (Sharifzadeh, Berrendorf, & Tresp, 2019) showed further improvements by including 3-D image information. (Tresp, Sharifzadeh, & Konopatzki, 2019) is a short paper that describes an earlier version of the model described in this paper.…”

Section: Cognitive Tensor Models and Related Modelsmentioning

confidence: 97%

The Tensor Brain: Semantic Decoding for Perception and Memory

Tresp,

Sharifzadeh,

Konopatzki

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We analyze perception and memory, using mathematical models for knowledge graphs and tensors, to gain insights into the corresponding functionalities of the human mind. Our discussion is based on the concept of propositional sentences consisting of subject-predicate-object (SPO) triples for expressing elementary facts. SPO sentences are the basis for most natural languages but might also be important for explicit perception and declarative memories, as well as intra-brain communication and the ability to argue and reason. Due to its compositional nature, a set of sentences can describe a scene in great detail, avoiding the explosion in complexity with flat representations. A set of SPO sentences can be described by a knowledge graph, which can be transformed into an adjacency tensor. We introduce tensor models, where concepts have dual representations as indices and associated embeddings, two constructs we believe are essential for the understanding of implicit and explicit perception and memory in the brain. We argue that a biological realization of perception and memory imposes constraints on information processing. In particular, we propose that explicit perception and declarative memories require a complex semantic decoder, which, in a basic realization, has four layers: First, a sensory memory layer, as a buffer for sensory input, second, a memoryless representation layer for the broadcasting of information -the "blackboard", or the "canvas" of the brain-, third, an index layer representing concepts, and fourth, a working memory layer as a processing center and data buffer. We discuss the operations of the four layers and relate them to the global workspace theory. Whereas simple semantic decoding might be performed already by higher animals, the generation of triple statements, requiring working memory as part of a complex semantic decoder, is a layered sequential process likely performed only by humans. In the resulting chatterbox decoding, semantic consistency is encouraged on the representation level. Both semantic and episodic memory contribute context and thus complement sensory input with non-perceptual information: agents have memory systems for a purpose, i.e., to make better decisions! In a Bayesian brain interpretation, semantic memory defines the prior distribution for observable triple statements. We propose that -in evolution and during development-semantic memory, episodic memory, and natural language evolved as emergent properties in agents' process to gain a deeper understanding of sensory information. Our mathematical model provides some fresh perspectives on much-debated issues concerning the relationship between perception, semantic memory, and episodic memory. We present a concrete model implementation and validate some aspects of our proposed model on benchmark data where we demonstrate state-of-the-art performance.

show abstract

Section: Cognitive Tensor Models and Related Modelsmentioning

confidence: 97%

The Tensor Brain: Semantic Decoding for Perception and Memory

Tresp,

Sharifzadeh,

Konopatzki

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…They described the spatial distribution of objects by using properties of regions, which contain positional relations, size relations, distance relations and shape relations. Moreover, Sharifzadeh et al [46] utilized 3D information in visual relation detection by synthetically generating depth maps using an RGB-to-Depth model incorporated within relation detection frameworks. They extracted pairwise feature vectors include depth, spatial, label and appearance.…”

Section: Multimodal Featuresmentioning

confidence: 99%

Scene Graph Generation: A Comprehensive Survey

Zhu¹,

Zhang²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and achieved good results. In this paper, we provide a comprehensive survey of recent achievements in this field brought about by deep learning techniques. We review 138 representative works that cover different input modalities, and systematically summarize existing methods of image-based SGG from the perspective of feature extraction and fusion. We attempt to connect and systematize the existing visual relationship detection methods, to summarize, and interpret the mechanisms and the strategies of SGG in a comprehensive way. Finally, we finish this survey with deep discussions about current existing problems and future research directions. This survey will help readers to develop a better understanding of the current research status and ideas.

show abstract

“…In VG, the distribution of labeled relations is highly imbalanced. Therefore, we additionally report Macro Recall (Sharifzadeh et al 2019;Chen et al 2019c) (mR@K) to reflect the improvements in the long tail of the distribution. In this setting, the overall recall is computed by taking the mean over recall per predicate.…”

Section: Gcn Vs Prior Model: a Matter Of Inductive Biasesmentioning

confidence: 99%

Classification by Attention: Scene Graph Classification with Prior Knowledge

Sharifzadeh

Baharlou

Tresp

2021

AAAI

Self Cite

View full text Add to dashboard Cite

A major challenge in scene graph classification is that the appearance of objects and relations can be significantly different from one image to another. Previous works have addressed this by relational reasoning over all objects in an image or incorporating prior knowledge into classification. Unlike previous works, we do not consider separate models for perception and prior knowledge. Instead, we take a multi-task learning approach by introducing schema representations and implementing the classification as an attention layer between image-based representations and the schemata. This allows for the prior knowledge to emerge and propagate within the perception model. By enforcing the model also to represent the prior, we achieve a strong inductive bias. We show that our model can accurately generate commonsense knowledge and that the iterative injection of this knowledge to scene representations, as a top-down mechanism, leads to significantly higher classification performance. Additionally, our model can be fine-tuned on external knowledge given as triples. When combined with self-supervised learning and with 1% of annotated images only, this gives more than 3% improvement in object classification, 26% in scene graph classification, and 36% in predicate prediction accuracy.

show abstract

Improving Visual Relation Detection using Depth Maps

Cited by 5 publications

References 0 publications

The Tensor Brain: Semantic Decoding for Perception and Memory

The Tensor Brain: Semantic Decoding for Perception and Memory

Scene Graph Generation: A Comprehensive Survey

Classification by Attention: Scene Graph Classification with Prior Knowledge

Contact Info

Product

Resources

About