A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective

Abstract

Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (e.g., social network analysis and recommender systems), computer vision (e.g., object detection and point cloud learning), and natural language processing (e.g., relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, i.e., 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.

Explainer on GNN-based segmentation networks

“…Image segmentation aims to separate an image into several semantic meaningful regions by labeling each pixel, depending on the object's appearance [15]. Deep neural networks have significantly improved this field.…”

“…Finally, GNN is a fast-growing field, and this paper can only discuss limited aspects of GNNs. In addition, a more comprehensive review of GNN is given by Chen et al [15].…”

Attention-Based Graph Neural Network for Label Propagation in Single-Cell Omics

“…For a hierarchical data structure such as a graph, graph neural networks (GNN) are well suited to perform learning [14,15]. GNNs have been shown to be effective at tasks such as node classification, link prediction, and graph classification, and have been applied to a wide range of domains including computer vision, natural language processing, electrical engineering, and bioinformatics [16][17][18][19].…”

ViGEO: an Assessment of Vision GNNs in Earth Observation

“…In such cases, a simple graph structure is imposed a-priori (e.g., based on distances) [12] or is automatically infererred by the neural network [13]. Few works investigated the application of GNNs to the vision domain for different tasks, mainly related to point clouds [14], with the Vision GNN architecture [15] (ViG) being the most successful architecture in image classification, achieving higher performances in the image classification task compared to the ViT architecture [10].…”