TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding

He, Dailan; Zhao, Yusheng; Jian, Luo; Hui, Tianrui; Huang, Shaofei; Zhang, Aixi; Liu, Si

doi:10.1145/3474085.3475397

Cited by 45 publications

(17 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the sparse, noisy, and limited semantic information of point clouds compared to 2D images make it difficult to accurately locate a referred object [36]. Additionally, the proximity of the referent to adjacent objects in the scene can lead to localization errors [2,1,4,40], and view-dependent descriptions can result in poor localization performance for referent localization based on spatial terms [15,36,40,14,13,39,12]. There are also localization errors when locating a unique referent among multiple visually similar objects [2,21,15,36,40,14,13,39,12,19,1,4].…”

Section: Non-interactive Visual Groundingmentioning

confidence: 99%

“…Additionally, the proximity of the referent to adjacent objects in the scene can lead to localization errors [2,1,4,40], and view-dependent descriptions can result in poor localization performance for referent localization based on spatial terms [15,36,40,14,13,39,12]. There are also localization errors when locating a unique referent among multiple visually similar objects [2,21,15,36,40,14,13,39,12,19,1,4]. Our approach introduces a new task of 3D visual grounding in a humanin-the-loop-based scenario, where body gestures are integrated into the scene to mitigate localization errors resulting from sparse, noisy, and semantically limited point clouds, object proximity, difficulty in distinguishing a unique referent among visually similar objects, and view-dependent descriptions.…”

Section: Non-interactive Visual Groundingmentioning

confidence: 99%

See 1 more Smart Citation

ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding

Lu¹,

Pei²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Aiming to link natural language descriptions to specific regions in a 3D scene represented as 3D point clouds, 3D visual grounding is a very fundamental task for human-robot interaction. The recognition errors can significantly impact the overall accuracy and then degrade the operation of AI systems. Despite their effectiveness, existing methods suffer from the difficulty of low recognition accuracy in cases of multiple adjacent objects with similar appearances. To address this issue, this work intuitively introduces the humanrobot interaction as a cue to facilitate the development of 3D visual grounding. Specifically, a new task termed Embodied Reference Understanding (ERU) is first designed for this concern. Then a new dataset called ScanERU is constructed to evaluate the effectiveness of this idea. Different from existing datasets, our ScanERU is the first to cover semi-synthetic scene integration with textual, real-world visual, and synthetic gestural information. Additionally, this paper formulates a heuristic framework based on attention mechanisms and human body movements to enlighten the research of ERU. Experimental results demonstrate the superiority of the proposed method, especially in the recognition of multiple identical objects. Our codes and dataset 1 are ready to be available publicly.

show abstract

Section: Non-interactive Visual Groundingmentioning

confidence: 99%

Section: Non-interactive Visual Groundingmentioning

confidence: 99%

ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding

Lu¹,

Pei²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…ReferIt3DNet [2] utilizes a graph convolutional network with input objects as nodes of the graph. 3DRefTransformer [1], LanguageRefer [30] TransRefer [16], and SAT [37] are Transformerbased methods that operate on language and 3D object point clouds. 3DRefTransformer [1] is an end-to-end Transformer model that incorporates an object pairwise spatial relation loss.…”

Section: Related Workmentioning

confidence: 99%

“…LanguageRefer [30] uses a Transformer architecture over bounding box embeddings and language embedding from DistilBert [31]. TransRefer [16] utilizes a Transformerbased network to extract entity-and-relation-aware represen-…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

Hsu¹,

Mao²

2023

Preprint

View full text Add to dashboard Cite

Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipulation. However, the variability of the 3D domain induces two fundamental challenges: 1) the expense of labeling and 2) the complexity of 3D grounded language. Hence, essential desiderata for models are to be data-efficient, generalize to different data distributions and tasks with unseen semantic forms, as well as ground complex language semantics (e.g., view-point anchoring and multi-object reference). To address these challenges, we propose NS3D, a neuro-symbolic framework for 3D grounding. NS3D translates language into programs with hierarchical structures by leveraging large language-to-code models. Different functional modules in the programs are implemented as neural networks. Notably, NS3D extends prior neuro-symbolic visual reasoning methods by introducing functional modules that effectively reason about high-arity relations (i.e., relations among more than two objects), key in disambiguating objects in complex 3D scenes. Modular and compositional architecture enables NS3D to achieve state-of-the-art results on the ReferIt3D view-dependence task, a 3D referring expression comprehension benchmark. Importantly, NS3D shows significantly improved performance on settings of data-efficiency and generalization, and demonstrate zero-shot transfer to an unseen 3D question-answering task.

show abstract