Developing a Hand Gesture Recognition System for Mapping Symbolic Hand Gestures to Analogous Emojis in Computer-Mediated Communication

Abstract

Recent trends in computer-mediated communication (CMC) have not only led to expanded instant messaging through the use of images and videos but have also expanded traditional text messaging with richer content in the form of visual communication markers (VCMs) such as emoticons, emojis, and stickers. VCMs could prevent a potential loss of subtle emotional conversation in CMC, which is delivered by nonverbal cues that convey affective and emotional information. However, as the number of VCMs grows in the selection set, the problem of VCM entry needs to be addressed. Furthermore, conventional means of accessing VCMs continue to rely on input entry methods that are not directly and intimately tied to expressive nonverbal cues. In this work, we aim to address this issue by facilitating the use of an alternative form of VCM entry: hand gestures. To that end, we propose a user-defined hand gesture set that is highly representative of a number of VCMs and a two-stage hand gesture recognition system (trajectory-based, shape-based) that can identify these user-defined hand gestures with an accuracy of 82%. By developing such a system, we aim to allow people using low-bandwidth forms of CMCs to still enjoy their convenient and discreet properties while also allowing them to experience more of the intimacy and expressiveness of higher-bandwidth online communication.

User-Defined Hand Gesture Interface to Improve User Experience of Learning American Sign Language

“…Besides, conventional means of accessing visual communication markers (VCM) rely on input entry methods that are not directly and intimately tied to expressive nonverbal cues. Koh et al [14] addressed this issue, by facilitating the use of an alternative form of VCM entry: hand gestures. Moreover, to fill this gap Dai et al [7] presented a training system, called CAPG-MYO, for userdefined hand gesture interaction.…”

Towards Augmented Reality Driven Human-City Interaction: Current Research on Mobile Headsets and Future Challenges

“…An earlier survey regarding smartglasses input defines four categories for classifying input techniques on smartglasses [1]. These categories are: on-device interaction [64], on-body interaction [75] [95], hands-free interaction (e.g., gaze [88] and whole-body gestures [93] [96] for emoji inputs [140]), and freehand interaction with wearable cameras [86] or sensors inside the closed environment of a vehicle [120], showing clear boundaries among the four categories. Nevertheless, as reflected in the most recent works, these boundaries are becoming more blurred.…”

“…The coexistence of tactile feedbacks and on-skin devices Designing tactile feedbacks with epidermal devices [140] On-body Computer vision (Headset) Similarity between body gestures and emjois Emjois input [72] On-body Tactile Alternative user perceptions other than visual and audio loads Feel-through feedbacks on sensitive skin surface [98] On-body Tactile Alternative user perceptions other than visual and audio loads Feel-through feedbacks on skin hair [84] On-body Radar Micro-gestures between fingers of one hand Gesture-to-command [78] On-body Infrared Thumb-to-fingertip microgestures (e.g., circle, triangle, rub, etc.) Gesture-to-command [86] On-body Computer vision (Wrist-worn) Micro-gestures between fingers of one hand Gesture-to-command [120] On-body Computer vision Intuitive Hand Gestures for controlling in-vehicle interfaces Gesture-to-command [171] On-body Computer vision Learning new hand gestures Gesture-to-command [71] On-body Electrodes and circuits Touch on sensitive and spacious skin surface, accomodating various gestures Gesture-to-command [81] On-body Near-field Communication (NFC) Enabling the spacious skin surface on human body as control devices Gesture-to-command [99] On-body RFID Micro-gestures driven by thumb-to-fingertip interaction Gesture-to-command [91,92] On-body Electrodes and circuits Touch events between two hands of two people In-city social events by Interpersonal Interaction [69] On-body Fluidic material Alternative user perceptions other than visual and audio loads Information display and notifications [85] On-body LED lights The spacious skin surface becomes swift and easy-to-reach channels for notifications Information display and notifications [80] On-body Touch-sensitive surface Miniature-size interface on a nail and fingertip-to-nail interaction…”

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

“…In recent years, the research of emoji has attracted more and more attention. Emojis are studied in some computermediated communications [29], such as twitter [30,31] and posts [32][33][34].…”