This paper proposes a real-time dynamic hand gesture recognition system based on Hidden Markov Models with incremental learning method (IL-HMMs) to provide natural human-computer interaction. The system is divided into four parts: hand detecting and tracking, feature extraction and vector quantization, HMMs training and hand gesture recognition, incremental learning. After quantized hand gesture vector being recognized by HMMs, incremental learning method is adopted to modify the parameters of corresponding recognized model to make itself more adaptable to the coming new gestures. Experiment results show that comparing with traditional one, the proposed system can obtain better recognition rates.