Abstract. In order to detect and count pedestrians in different kinds of scenes, this paper put forward a method of solving the problem on video sequences captured from a fixed camera. After preprocessing operations on the original video sequences (Gaussian mixture modeling, three-frame-differencing, image binaryzation, Gaussian filtering, dilation and erosion) we extract the relatively complete pedestrian contours. Then we use the least square ellipse fitting method on those contours that has been extracted, the center of the ellipse is undoubtedly regarded as the tracking point of a pedestrian. With those points, a pedestrian matching pursuit and counting algorithm based on object motion continuity is used for tracking and counting pedestrians, this method can be better used in those scenes which are sparse and rarely obscured. Experiments validate that our pedestrian matching pursuit and counting algorithm has obvious superiorities: good real-time performance and high accuracy.