This paper constructs a sports action recognition model based on deep learning (DL) and clustering extraction algorithm. For the input detection image frame, athletes’ movements are detected through DL network, and then athletes’ sports movements are fused. Moreover, it expands new knowledge and improves learning ability through automatic learning training set. The neural network (NN) is applied to the sample set containing images of nonathletes, and the negative training sample set is iteratively enhanced according to the generated false positives, and the results are optimized by clustering method. Simulation experiments show that compared with other algorithms, the clustering extraction algorithm in this paper has achieved superior performance in recognition rate and false alarm rate, and the recognition speed is faster. The aim is to extract the athletes’ training postures through the analysis of sports movements, so as to assist coaches to train athletes more professionally and provide some reference for sports movement recognition.