Predicting and understanding pedestrian intentions is crucial for autonomous vehicles and mobile robots to navigate in a crowd. However, the movement of pedestrian is random. Pedestrian trajectory modeling needs to consider not only the past movement of pedestrians, the interaction between different pedestrians, the constraints of static obstacles in the scene, but also multi-modal of the human trajectory, which brings challenges to pedestrian trajectory prediction. Most of the existing trajectory prediction methods only consider the interaction between pedestrians in the scene, ignoring the static obstacles in the scene can also have impacts on the trajectory of pedestrian. In this paper, a scalable relative interactive spatial-temporal graph generation adversarial network architecture (RISTG-GAN) is proposed to generate a reasonable multi-modal prediction trajectory by considering the interaction effects of all agents in the scene. Our method extends recent work on trajectory prediction. First, LSTM nodes are flexibly used to model the spatial-temporal graph of human-environment interactions, and the spatial-temporal graph is converted into feed-forward differentiable feature coding, and the time attention module is proposed to capture the trajectory information in time domain and learn the time dependence in long time range. Then, we capture the relative importance of the interaction of all agents in the scene on the pedestrian trajectory through the improved relative scaled dot product attention and use the generative adversarial network architecture for training to generate reasonable pedestrian future trajectory distribution. Experiments on five commonly used real public datasets show that RISTG-GAN is better than previous work in terms of reasoning speed, accuracy and the rationality of trajectory prediction.INDEX TERMS Pedestrian trajectory prediction, spatial-temporal graph, time attention, relative scaled dot product attention, generative adversarial network.