To recognize the actions of human's lower limbs accurately and quickly, a novel action recognition method based on a human joint was proposed. Firstly, hip joint was chosen as the recognition object, its y coordinates were as recognition parameter, and human action characteristics were achieved based on Butterworth filter and wavelet transform. Secondly, an improved self-organizing competitive neural network was proposed, which could classify the action characteristics automatically according to the classification number. The classification results of motion capture data proved the validity of the neural network. Finally, an action recognition method based on hidden Markov model (HMM) was introduced to realize the recognition of classification results of human action characteristics with the change direction of y coordinates. The proposed action recognition method needs less action information and has a fast calculation speed. Experiments proved the method had a high recognition rate and a good application prospect.