Remote control for the position and status of a machine or an equipment can often be teleoperated by gestures in an intelligent manufacturing environment. In order to solve the problems that gestures with two directions such as left and right cannot be detected by single ultrasonic frequency, double different ultrasonic frequencies are used to detect gestures by the Doppler shift, and an algorithm of the recognition gesture based on the DAG‐SVMs mixed Hidden Markov Model (HMM) is proposed to identify and classify the extracted feature sequences. Thus, four more types of gestures are expanded other than that of reading display screen information, and the comparative experiments to classify and recognise gestures of teleoperation are made with DAG‐SVMs, the HMM, the DAG‐SVMs mixed HMM, and other improved HMM algorithms. The test results have shown that the mean rate of gesture recognition for the algorithm based on the DAG‐SVMs mixed HMM is 94.917%, which is 9.497% higher than that of the unimproved HMM, and its recognition accuracy of complex teleoperation gestures is improved by 2.3% compared with other improved HMM algorithms. The experimental results show that the DAG‐SVMs mixed HMM algorithm has a good effect on recognition for the gestures of teleoperation and it can perform gesture recognition accurately.