There has been a rapid increase in the number of Internet of Things (IoT) devices in the last few years, providing a wide range of services such as camera feeds, light controls, and door locks for remote access. Access to IoT devices, whether within the same environment or remotely via the Internet, requires proper security mechanisms in order to avoid disclosing any secure information or access privileges. Authentication, on which other security classes are built, is the most important part of IoT security. Without ensuring that the authorized party is who it claims to be, other security factors would be useless. Additionally, with the increased mobility of IoT devices, traditional authentication mechanisms, such as a username and password, are less effective. Numerous security challenges in the IoT domain have resulted in the proposal of many different approaches to authentication. Many of these methods require either carrying an authentication token, such as a smartcard, or restricting access to a particular physical location. Considering that most IoT devices contain a wide array of sensors, a large amount of contextual information can be provided. Thus, real-time security mechanisms can protect user access by, for example, utilizing contextual information to validate requests. A variety of contextual information can be retrieved to strengthen the authentication process, both at the time of access request and throughout the entire access session, without requiring user interaction, which avoids the risk of being discovered by attackers of these features. In this paper, we introduce a continuous authentication framework that integrates contextual information for user authentication in smart homes. The implementation and evaluation show that the framework can protect smart devices against unauthorized access from both anonymous and known users, either, locally or remotely, in a flexible manner and without requiring additional user intervention.