Deep learning technology performs effect in feature extraction of images. Nowadays, with the development of video monitoring, the application of deep learning technology to surveillance video has profound implications. The effects of traditional video recognition are not satisfactory, but deep learning methods perform effect in many scenes of image classification. This paper proposed a novel object detection algorithm in video. It combined the traditional methods of extracting feature and deep learning algorithm to realize vehicle identification based on surveillance video. The method used the frame difference method and background subtraction to preprocess the image, and then trained a network model based on YOLO to perform object detection and obtain the categories and location information of the monitored vehicle. Compared with the existing object detection algorithms faster RNN, our method can achieve higher accuracy and can significantly shorten the time for detection, which can recognize the object of vehicle video quickly and efficiently. The method can meet the requirements of real-time detection.