Object tracking is the problem of estimating the positions of moving objects in image sequences, which is significant in various applications. In traffic surveillance, the tasks of tracking and recognition of moving objects are often inseparable and the accuracy and reliability of a surveillance system can be generally enhanced by integrating them. In this paper, we proposed a traffic surveillance system that features of classification of pedestrian and vehicle types while tracking, which works well in challenging real-word conditions. The object tracking is implemented by the Mean Shift and object classification is implemented with several different classification algorithms including k-nearest neighborhood (kNN), support vector machine (SVM), multi-layer perceptron (MLP), and random forest (RF), with high classification accuracies.