Relationformer: A Unified Framework for Image-to-Graph Generation

Abstract

A comprehensive representation of an image requires understanding objects and their mutual relationship, especially in image-to-graph generation, e.g., road network extraction, blood-vessel network extraction, or scene graph generation. Traditionally, image-to-graph generation is addressed with a two-stage approach consisting of object detection followed by a separate relation prediction, which prevents simultaneous object-relation interaction. This work proposes a unified onestage transformer-based framework, namely Relationformer that jointly predicts objects and their relations. We leverage direct set-based object prediction and incorporate the interaction among the objects to learn an object-relation representation jointly. In addition to existing [obj]-tokens, we propose a novel learnable token, namely [rln]-token. Together with [obj]-tokens, [rln]-token exploits local and global semantic reasoning in an image through a series of mutual associations. In combination with the pair-wise [obj]-token, the [rln]-token contributes to a computationally efficient relation prediction. We achieve state-of-the-art performance on multiple, diverse and multi-domain datasets that demonstrate our approach's effectiveness and generalizability. 2 * equal contribution 2 code is available at https://github.com/suprosanna/relationformer

Patched Line Segment Learning for Vector Road Mapping

“…Road Graph Representations. There have been plenty of studies for vector road mapping, mainly relying on either the rasterized road map or the keypoint/vertex-based graph representations, and derived two categories, the segmentationbased (Máttyus, Luo, and Urtasun 2017;Zhou, Zhang, and Wu 2018;Mei et al 2021;Wang et al 2023;Batra et al 2019;Cheng et al 2021) and the keypoint-based approaches (He et al 2020;He, Garg, and Chowdhury 2022;Shit et al 2022;Yang et al 2023;Xie et al 2023). Regarding the popularity of end-to-end learning for better performance, the state-of-the-art approaches (He, Garg, and Chowdhury 2022;Xu et al 2023b) mainly learn keypoints (i.e., graph vertices) and the connectivity between vertices while using the rasterized road masks/maps as the additional supervision signals to enhance the feature representation ability of ConvNets.…”

MolNexTR: A Generalized Deep Learning Modelfor Molecular Image Recognition

“…Second, in the encoder stage, We propose a novel combined CNN and ViT molecular image encoder network that is aware of both local atom information representation and long-range interatomic feature dependencies in the learning process. Third, in the decoder stage, we combine the Pix2seq and Relationformer architecture [14,15], using an autoregressive decoder to predict atoms and their coordinates as a sequence and predict the chemical bonds between the atoms, composing the 2D molecular graph. Finally, we include chemical knowledge as symbolic constraints to the model, such as determining the chirality of atoms from the predicted pattern.…”