Abstract-Face recognition system have now been widely adopted in wide range of applications starting from mobile authentication to attendance system. Due to availability of the camera with various devices, and ease of development and deployment of such system. The accuracy of many of the modern day commercial and research face recognition system are as per the other biometrics like fingerprint biometric. Alongside face biometric systems, video based face recognition systems are also gaining popularity every day in wide range of applications starting from content serving to surveillance. Such techniques are affected by uncontrolled environment, pose variation, light intensity variations, motion artefacts, and blurring and high frame rate. Therefore existing isolated facial biometric systems do not work well in the context of video based face recognition system. Such a systems are computationally expensive and demand a fast recognition technique along with accuracy as detection and recognition task need to be performed on every frame. Though some of the past research works have proposed different techniques in video based face recognition systems, they are yet not being as widely standardized and accepted as isolated face biometric system. This paper proposes a novel technique for real time face recognition using local directional pattern (LDP). Our method selects a frame as reference frame and extracts the face in the frame which is recognized by matching the database frames. Faces in the subsequent frames are matched with the reference frame only, reducing the number of comparison significantly. When the face in a video frame is found to not matching with the current reference frame, it is matched against the database frames and the recognized face is considered as the current reference face. We validate our technique by constructing random video sequences of one minute at 2fps using Yale face dataset. The techniques are validated against two popular video based face recognition techniques: Ensemble based technique and Exemplar based technique and a traditional isolated face recognition technique in the form of PCA based face biometric. Results show that the proposed technique outperforms the other peers in terms of both speeds of recognition as well as recognition accuracy.Keyword -Face Recognition, LBP, LDP, Yale I. INTRODUCTION For over three decades, human face recognition has been an active area of research. For face based biometric authentication scheme, traditionally face recognition has been considered where the responsibility has been to match a captured face against sets of faces stored in the database [1]. In 1991 it was proposed [2], Starting from popular Eigen face based face recognition technique, many algorithms are being proposed which focuses on solving specific challenges in face recognition like pose invariance [3], illumination invariance [4], partial face recognition [5], expression invariance [6], face recognition in night [7] and so on. Focus on unconstrained face recognition [8] most of t...