Pedestrian tracking and detection of crowd abnormal activity under dynamic and complex background using Intelligent Video Surveillance (IVS) system are beneficial for security in public places. This paper presents a pedestrian tracking method combing Histogram of Oriented Gradients (HOG) detection and particle filter. This method regards the particle filter as the tracking framework, identifies the target area according to the result of HOG detection and modifies particle sampling constantly. Our method can track pedestrians in dynamic backgrounds more accurately compared with the traditional particle filter algorithms. Meanwhile, a method to detect crowd abnormal activity is also proposed based on a model of crowd features using Mixture of Gaussian (MOG). This method calculates features of crowd-interest points, then establishes the crowd features model using MOG, conducts self-adaptive updating and detects abnormal activity by matching the input feature with model distribution. Experiments show our algorithm can efficiently detect abnormal velocity and escape panic in crowds with a high detection rate and a relatively low false alarm rate.