Trajectories of moving objects provide crucial clues for video event analysis especially in surveillance applications. In this paper, we proposed a novel approach for detecting abnormal events in video surveillance. Our approach is based on trajectory analysis involving two phases. In the first phase, we extracted clusters of normal events through an agglomerative hierarchical clustering of saved trajectories that were of different lengths, of different local time shifts and containing noise. Then, for each cluster a model was established. In the second phase, we aimed to classify a new event as normal or abnormal one. To achieve this objective, a comparison was performed with the extracted clusters' models thereby reducing the complexity and accelerating the classification process. Experiments were conducted to demonstrate the efficacy and the performance of our approach.