In order to improve the recognition accuracy of human falling actions, the impact of randomness of actions is reduced. To this end, this paper proposes an automatic recognition method for physical fitness human fall based on pose data sequence. The color camera is used to collect the fall motion images of the physical fitness personnel, and the motion image preprocessing is completed by extracting the fall motion features of the human body, tracking and adjusting the fall motion of the human body. A model of human body fall movement displacement feature extraction from posture data sequence is constructed. By tracking the displacement feature points, the automatic recognition of physical fitness human body fall movement based on posture data sequence is realized. The experimental results confirm that the proposed method can effectively obtain the details of the fall motion images of physical fitness. When the number of human falling actions reaches 500, the accuracy of action recognition is also as high as 77%, improving the recognition effect of human fall action.