Due to the lack of particular algorithms for automatic detection and tracking of person face(s), we have developed a new algorithm to achieve detection and single/multiple face tracking in different background video sequence. To detect faces, skin sections are segmented from the frame by means of YCbCr color model; and facial features are used to agree whether these sections contain person face or not. This procedure is challenging, because face color is unique and some objects may have similar color. Further, color and Eigen features are extracted from detected faces. Based on the points detected in facial region, point tracker tracks the user specified number of faces throughout the video sequence. The developed algorithm was tested on challenging dataset videos; and measured for performance using standard metrics. Test results obtained ensure the efficiency of proposed algorithm at the end.