Vehicle detection and classification plays an important role in intelligent transportation system. Compared with traditional detectors, the detection and classification based on traffic surveillance video shows a huge advantage in its flexibility and continuity. However, to get wide applicability and strong robustness, most current methods focus on improving the accuracy of detectors by adjusting network parameters constantly, or increasing the size of training sets, which challenges the collection and labeling of data, the performance of computers, the scope of application and so on. Moreover, the unique continuity characteristic of the video, which can be used to describe the motion features of vehicle, is often ignored. Take these facts into account, this paper proposed a video-based vehicle detection and classification method, which is based on static appearance features and motion features both. Four detectors of different performance were trained with small training sets, and the designed algorithms for the remove, selection and reorganization of detected objects contribute to obtaining the optimal results of detection and classification. The experiment results show that the proposed method is able to detect and classify vehicles with more than 0.95 accuracy dealing with different road environments.