Aiming at the related problems existing in the field of leisure sports computing, in order to study the behavior recognition of leisure sports by deep residual network, based on the deep residual neural network theory, the behavior recognition algorithm and the corresponding robust model are used to analyze the leisure sports related samples, and the correlation model is used to predict and analyze the leisure sports related content. The results show that the change curves of Sig and Tanh functions can be divided into slow increasing stage, linear increasing stage, and stable stage. The y value corresponding to ReLU curve shows a linear change trend with the increase of x value. The Leaky function’s corresponding curve can be divided into two stages. The function coincides with the ReLU function in the first quadrant and remains linear in the third quadrant. The activation function curves corresponding to layers 56 and 20 have a relatively large variation range, and both of them show an overall trend of gradual decline. On the whole, the curve value corresponding to layer 56 is higher than that corresponding to layer 20, indicating that the method of layer 20 is relatively good and the corresponding training error is relatively low. It can be seen from the robustness recognition rate of various methods under different training samples that Fl has the highest overall data recognition rate while Sc has relatively poor stability. However, the recognition rate of IDCC and DCC shows a relatively flat trend, indicating that these two methods have certain advantages in describing the robust recognition rate. The research results can provide theoretical support for the application of deep residual neural networks in other fields.