Recently, in the surveillance field, scene analysis research is a hot topic, and many useful algorithms are developed. They not only reduce human power, but also make surveillance in real time. In this paper, we propose a robust and efficient anomaly-detection algorithm in traffic surveillance system. The proposed method consists of two parts:(1) scene modeling part, and (2) anomaly detection part. First part, to systemically collect trajectories of moving objects, we apply the sparse optical flow method to foreground regions extracted by a conventional background modeling method. These collected trajectories are represented as 3-dimensional feature vectors whose components are x and y coordinates and moving direction, and they are clustered by k-means clustering method. After this process, all feature vectors are assigned clustering labels, and then we construct a trajectory histogram based on cells whose mean grid with a particular size to make the scene model. Then we apply the entropy concept to generated histograms in order to handle some regions where the uncertainty of motion pattern is high. In the anomaly detection part, we get features of objects in a image and track them with the same way in the scene modeling part. At this time, they are classified by nearest neighborhood method. From this result of classification, we can detect anomalies in the traffic video by comparing it with the scene model. Experimental results demonstrate that the anomaly detection rate of the proposed method is very high, and the processing speed is almost real time.