This work studies the features of a proposed automated stroke self-screening application that utilizes the gyroscope and accelerometer devices in smartphones to determine the possible onset of a stroke by assessing arm muscle weakness. The application requires users to perform two arm movements to evaluate arm weakness and pronation: Curl-up and Raise-up. For the purpose of the study, 68 subjects, consisting of 36 stroke patients with symptoms of arm weakness and 32 healthy subjects, consented to participate. A total of 78 handcrafted features were proposed, 26 of which were extracted from Curl-up and Raise-up for each arm. Then, the differences between corresponding features for each arm were calculated. These features were then tested on 63 combinations of three classical feature selection methods, three feature sets (i.e., Curl-up-only features, Raise-up-only features, and both-exercises combined features) and seven well-known classification methods. The results from ten runs of 10-fold cross-validation showed that Curlup-only features achieved an average sensitivity of 83.3%, significantly higher than those of the Raise-uponly features or both-exercises features. From all possible combinations, the random forest classification based on information gain feature selection from Curl-up-only features achieved the most efficient results for arm-weakness-stroke screening. It achieved an average sensitivity of 94.8%, an average specificity of 75.2%, an average accuracy of 84.1%, and an average area under the receiver operating characteristic curve of 85.0%. Our work proposes a novel accessible method to screen symptoms of arm weakness that may indicate the onset of a stroke using a single mobile device. In the future, we can combine this method with other methods of evaluating facial drooping and slurred speech to create a complete Face, Arm, Speech, Time (FAST) assessment application.