Static Authentication provides a secure framework for a one-time authentication session, but fails to authenticate the user throughout the session. This presents the possibility of an imposter gaining access when a user session is active and the user moves away from the system. The goal of continuous authentication is to authenticate the user right from the initial stages of log-in till log-out. Intuitively, this can be implemented by extrapolating the tried-and-tested static authentication techniques throughout the session. However, extrapolating one-time authentication techniques poses new challenges of being computationally expensive, restricting the user's movement and postures in front of the system, depending on extra expensive hardware and deviating the user from his workflow. In these situations, the user no longer remains uninterrupted by the authentication process in the background. The proposed framework provides unobtrusive Continuous Authentication, by alternating between two modes which utilize hard and soft biometrics respectively, depending on certain confidence parameters. We use facial features as the hard biometric trait for recognizing the user. Employing face recognition for extended periods of time produces noise, which is dampened by using a supervised machine learning algorithm. The color of user's clothing as the soft biometric trait relieves the CPU of comparatively high computation and relaxes constraints on the user's upper body movement.