Continuous authentication has been proposed as a possible approach for passive and seamless user authentication, using sensor data comprising biometric, behavioral, and context-oriented characteristics. Since these are personal data being transmitted and are outside the control of the user, this approach causes privacy issues. Continuous authentication has security challenges concerning poor matching rates and susceptibility of replay attacks. The security issues are mainly poor matching rates and the problems of replay attacks. In this survey, we present an overview of continuous authentication and comprehensively discusses its different modes, and issues that these modes have related to security, privacy, and usability. A comparison of privacy-preserving approaches dealing with the privacy issues is provided, and lastly recommendations for secure, privacy-preserving, and user-friendly continuous authentication.