ST‐SIGMA: Spatio‐temporal semantics and interaction graph aggregation for multi‐agent perception and trajectory forecasting

Abstract

Scene perception and trajectory forecasting are two fundamental challenges that are crucial to a safe and reliable autonomous driving (AD) system. However, most proposed methods aim at addressing one of the two challenges mentioned above with a single model. To tackle this dilemma, this paper proposes spatio‐temporal semantics and interaction graph aggregation for multi‐agent perception and trajectory forecasting (ST‐SIGMA), an efficient end‐to‐end method to jointly and accurately perceive the AD environment and forecast the trajectories of the surrounding traffic agents within a unified framework. ST‐SIGMA adopts a trident encoder–decoder architecture to learn scene semantics and agent interaction information on bird’s‐eye view (BEV) maps simultaneously. Specifically, an iterative aggregation network is first employed as the scene semantic encoder (SSE) to learn diverse scene information. To preserve dynamic interactions of traffic agents, ST‐SIGMA further exploits a spatio‐temporal graph network as the graph interaction encoder. Meanwhile, a simple yet efficient feature fusion method to fuse semantic and interaction features into a unified feature space as the input to a novel hierarchical aggregation decoder for downstream prediction tasks is designed. Extensive experiments on the nuScenes data set have demonstrated that the proposed ST‐SIGMA achieves significant improvements compared to the state‐of‐the‐art (SOTA) methods in terms of scene perception and trajectory forecasting, respectively. Therefore, the proposed approach outperforms SOTA in terms of model generalisation and robustness and is therefore more feasible for deployment in real‐world AD scenarios.

Local feature matching and tracking optimization of machine VSLAM based on AG Sinkhorn

“…threshold is within the range of, 2,5 the matching accuracy of the four combinations shows a rapid increase, and it tends to stabilize when the judgment threshold is within the range of. 5,10 Figure 6 shows the accuracy recall curve. Figure 6A was tested on the HPatches Illumination dataset, while Figure 6B was tested on the HPatches View dataset.…”

“…The end‐to‐end LFM was implemented in view of DNN. In terms of matching, the AG neural network model is used to search for the optimal matching matrix through the sinkhorn algorithm, achieving high‐quality image matching 10 . The basic framework of LFM is shown in Figure 3.…”

“…Moreover, AG Sinkhorn does not require manual intervention when classifying and segmenting images, and the extracted feature points can be directly used for subsequent feature matching (FM) 7,8 . However, the current VSLAM system suffers from an issue of excessive matching of some feature points, which leads to low real‐time performance of the system, as well as inaccurate matching of some feature points, resulting in significant errors in the system 9,10 . The key technologies of VSLAM based on DL are studied, and on the basis of obtaining feature point calibration dataset based on self‐supervised network: Vector of locally aggregated descriptors based on neural network.…”

Virtual reality modeling application based on multi perspective and deep learning in the new media presentation and brand building of Dongguan City memory

“…It is a common method for VR platforms and has high efficiency. 29 In the VR platform, building terrain using the altitude map includes four steps: importing the altitude map, setting parameters, and generating terrain and texture maps. Figure 6 shows the specific process of terrain construction.…”

“…To ensure efficiency, the height map is used as the data basis. It is a common method for VR platforms and has high efficiency 29 . In the VR platform, building terrain using the altitude map includes four steps: importing the altitude map, setting parameters, and generating terrain and texture maps.…”