With the development of science and technology, manipulators are increasingly used in industrial production and daily life. The manipulator with traditional control mode has high requirements for operators, complicated operation, and human-computer interaction is inconvenient. In this paper, an intelligent manipulator system based on gesture control is designed. FPGA is used as the control core of the system. We apply the improved Canny algorithm processing, which can automatically recognize gesture information. After judgment, the control signal is sent to the manipulator to control the manipulator to complete the specified action. Through experimental testing, the system can successfully recognize all kinds of gestures, and can well control the manipulator to complete all kinds of actions. The human-computer interaction is more real, which greatly improves the intelligence of the manipulator system and helps to improve the work efficiency.