With the rapid development of deep learning algorithms, it is gradually applied in UAV (Unmanned Aerial Vehicle) driving, visual recognition, target tracking, behavior recognition, and other fields. In the field of sports, many scientists put forward the research of target tracking and recognition technology based on deep learning algorithms for athletes’ trajectory and behavior capture. Based on the target tracking algorithm, a regional proposal network RPN algorithm combined with the twin regional proposal network Siamese algorithm is proposed to study the tracking and recognition technology of athletes’ behavior. Then, the adaptive updating network is used to track the behavior target of athletes, and the simulation model of behavior recognition is established. This algorithm is different from the traditional twin network algorithm. It can accurately take the athlete’s behavior as the target candidate box in model training and reduce the interference of environment and other factors on model recognition. The results show that the Siamese-RPN algorithm can reduce the interference from the background and environment when tracking the athletes’ target behavior trajectory. This algorithm can improve the training behavior recognition model, ignore the background interference elements of the behavior image, and improve the accuracy and overall performance of the model. Compared with the traditional twin network method for sports behavior recognition, the Siamese-RPN algorithm studied in this paper can perform offline operations and distinguish the interference factors of athletes’ background environment. It can quickly capture the characteristic points of athletes’ behavior as the data input of the tracking model, so it has excellent popularization and application value.