The research content of this paper is mainly aimed at the automatic recognition of human behavior in video surveillance. It usually contains two main steps: foreground detection and behavior recognition. In the foreground detection step, this paper studied related algorithms and theories, briefly analyzed the principles and the advantages and disadvantages of various algorithms, finally choose the Vibe algorithm to extract the motive targets, and made some improvements to the original version of Vibe, which can get a more satisfied result. In behavior recognition step, we roughly classified the existing algorithms into three categories: template matching method, state space method, and semantic description method. After analyzing these three types of algorithms, we decided to select the Support Vector Machine (SVM) for training. We used the Star model for the feature collection. After the simulation experiment, the recognition result is not bad.