We are proposing a unique novel algorithm for tracking human face(s) in different background video sequences. In the beginning, Eigen features and corner points are extracted from the detected face(s). HOG (Histograms of Oriented Gradients) features are isolated from corner points. Eigen and HOG features are combined together. Using these combined features, point tracker keeps track of the face(s) in the frames of the video sequence. Proposed algorithm is being tested on challenging datasets video sequences with technical challenges such as partial occlusion (e.g. moustache, beard, spectacles, helmet, headscarf etc.), changes in expression, variations in illumination and pose; and measured for performance using standard metrics such as accuracy, precision, recall and specificity. Experimental results clearly indicate the robustness of the proposed algorithm on all different background challenging video sequences.