This paper proposed a new approach for object tracking and pattern recognition of moving objects during real time video streaming. This approach uses motion based multi-object movement techniques for tracking the objects. Moreover, Spectral clustering with Dynamic Time Warping (DTW) and Naïve Bayes method are used for pattern recognition of tracked objects. This system tracks the moving objects collected as a batch of videos then the pattern recognition technique uses for analyzing vehicles movement to determine normal or abnormal behavior. This paper proposes the tracking algorithm for all moving objects and pattern recognition for only moving vehicles. The performance of tracking trajectories is calculated by finding recall and precision values, which are greater than 95%. The experimental result shows that Naïve Bayes is better than spectral clustering for the classification of vehicle trajectories that conforms Naïve Bayes is an effective tool to scale the pattern recognition of moving vehicles.