Block matching motion estimation plays significant role for video processing and Computer Vision. Various methods based on mean shift and particle trackers are used to track the object on video sequence. Block matching algorithm is normally used for video compression technique. A novel, simple and fast block matching algorithm has been proposed using predictive motion vector for object tracking. Color histogram is used for matching criteria for the motion tracking. The proposed algorithm is considered with a flexible size of block as well as pixel displacement. Also early termination of the object as well as object out of boundary is considered. Experiments were conducted on different types of real world movie sequences.