There are different modes of interaction with a software keyboard on a smartphone, such as typing and swyping. Patterns of such touch interactions on a keyboard may reflect emotions of a user. Since users may switch between different touch modalities while using a keyboard, therefore, automatic detection of emotion from touch patterns must consider both modalities in combination to detect the pattern. In this paper, we focus on identifying different features of touch interactions with a smartphone keyboard that lead to a personalized model for inferring user emotion. Since distinguishing typing and swyping activity is important to record the correct features, we designed a technique to correctly identify the modality. The ground truth labels for user emotion are collected directly from the user by periodically collecting self-reports. We jointly model typing and swyping features and correlate them with user provided self-reports to build a personalized machine learning model, which detects four emotion states (happy, sad, stressed, relaxed). We combine these design choices into an Android application TouchSense and evaluate the same in a 3-week in-the-wild study involving 22 participants. Our key evaluation results and post-study participant assessment demonstrate that it is possible to predict these emotion states with an average accuracy (AUCROC) of 73% (std dev. 6%, maximum 87%) combining these two touch interactions only.

In Affective Computing, different modalities, such as speech, facial expressions, physiological properties, smartphone usage patterns, and their combinations, are applied to detect the affective states of a user. Keystroke analysis i.e. study of the typing behavior in desktop computer is found to be an effective modality for emotion detection because of its reliability, non-intrusiveness and low resource overhead. As smartphones proliferate, typing behavior on smartphone presents an equally powerful modality for emotion detection. It has the added advantage to run in-situ experiments with better coverage than the experiments using desktop computer keyboards. This work explores the efficacy of smartphone typing to detect multiple affective states. We use a qualitative and experimental approach to answer the question. We conduct an online survey among 120 participants to understand the typing habits in smartphones and collect feedback on multiple measurable parameters that affect their emotion while typing. The findings lead us to design and implement an Android based emotion detection system, TapSense, which can identify four different emotion states (happy, sad, stressed, relaxed) with an average accuracy (AUCROC) of 73% (maximum of 94%) based on typing features only. The analysis also reveals that among different features, typing speed is the most discriminative one.

No abstract

No abstract

The rate of mental health disorders is rising across the globe. While it significantly affects the quality of life, an early detection can prevent the fatal consequences. Existing literature suggests that mobile based sensing technology can be used to determine different mental health conditions like stress, bipolar disorder. In today's smartphone based communication, a significant portion is based on instant messaging apps like WhatsApp; thus providing the opportunity to unobtrusively monitor the text input interaction pattern to track mental state. We, in this paper, leverage on the text entry pattern to track multiple emotion states. We design, develop and implement an Android based smartphone keyboard EmoKey, which monitors user's typing pattern and determines four emotion states (happy, sad, stressed, relaxed) by developing an on-device, personalized machine learning model. We evaluate EmoKey with 22 participants in a 3-week in-the-wild study, which reveals that it can detect the emotions with an average accuracy (AUCROC) of 78%.