Smartphones are inescapable devices, which are becoming more and more intelligent and context-aware with emerging sensing, networking and computing capabilities. They offer a captivating platform to the users for performing a wide variety of tasks including socializing, communication, sending or receiving emails, storing and accessing personal data etc. at anytime and anywhere. Nowadays, loads of people tend to store different types of private and sensitive data in their smartphones including bank account details, personal identifiers, accounts credentials, and credit card details. A lot of people keep their personal e-accounts logged in all the time in their mobile devices. Hence these mobile devices are prone to different security and privacy threats and attacks from the attackers. Commonly used approaches for securing mobile devices such as passcode, PINs, pattern lock, face recognition, and fingerprint scan are vulnerable and exposed to several attacks including smudge attacks, side-channel attacks, and shoulder-surfing attacks. To address these challenges, a novel continuous authentication scheme is presented in this study, which recognizes smartphone users on the basis of their physical activity patterns using accelerometer, gyroscope, and magnetometer sensors of smartphone. A series of experiments are performed for user recognition using different machine learning classifiers, where six different activities are analyzed for the multiple locations of smartphone on the user's body. SVM classifier achieved the best results for user recognition with an overall average accuracy of 97.95%. A comprehensive analysis of the user recognition results validates the efficiency of the proposed scheme.