Sign sentence recognition with smart watches

Ekiz, Deniz; Kaya, Gamze Ege; Buğur, Serkan; Güler, Sila; Buz, Buse; Kosucu, Bilgin; Arnrich, Bert

doi:10.1109/siu.2017.7960255

Cited by 20 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since then, research has been conducted on applying different approaches and different devices in gesture-based SLR. In 2017, Ekiz et al [25] firstly attempted to capture the hand movements of signers with smart watches and used dynamic time warping (DTW) to compute the distances between the gestures and the templates in different dimensions for SLR.…”

Section: Related Workmentioning

confidence: 99%

A Portable Sign Language Collection and Translation Platform with Smart Watches Using a BLSTM-Based Multi-Feature Framework

2022

View full text Add to dashboard Cite

Continuous sign language recognition (CSLR) using different types of sensors to precisely recognize sign language in real time is a very challenging but important research direction in sensor technology. Many previous methods are vision-based, with computationally intensive algorithms to process a large number of image/video frames possibly contaminated with noises, which can result in a large translation delay. On the other hand, gesture-based CSLR relying on hand movement data captured on wearable devices may require less computation resources and translation time. Thus, it is more efficient to provide instant translation during real-world communication. However, the insufficient amount of information provided by the wearable sensors often affect the overall performance of this system. To tackle this issue, we propose a bidirectional long short-term memory (BLSTM)-based multi-feature framework for conducting gesture-based CSLR precisely with two smart watches. In this framework, multiple sets of input features are extracted from the collected gesture data to provide a diverse spectrum of valuable information to the underlying BLSTM model for CSLR. To demonstrate the effectiveness of the proposed framework, we test it on an extremely challenging and radically new dataset of Hong Kong sign language (HKSL), in which hand movement data are collected from 6 individual signers for 50 different sentences. The experimental results reveal that the proposed framework attains a much lower word error rate compared with other existing machine learning or deep learning approaches for gesture-based CSLR. Based on this framework, we further propose a portable sign language collection and translation platform, which can simplify the procedure of collecting gesture-based sign language dataset and recognize sign language through smart watch data in real time, in order to break the communication barrier for the sign language users.

show abstract

Section: Related Workmentioning

confidence: 99%

A Portable Sign Language Collection and Translation Platform with Smart Watches Using a BLSTM-Based Multi-Feature Framework

2022

View full text Add to dashboard Cite

show abstract

“…These previous works use devices such as RGB cameras [5,21,27,29,33,35,46,54,55], motion sensors (e.g., Leap Motion) [14,41], depth cameras/sensors (e.g., Kinect) [6,10,11,16,38,48,51], or electromyogram (EMG) sensors [53,57] to capture user hand motions and combine sensing results with various machine learning models to infer the word being expressed. More recently, research has considered the contextual meanings of words and their syntaxial relationships to generate proper sentences from sign language motions [13,14,21]. However, it is not trivial to apply these technologies in everyday situations since they either require additional devices or infrastructure support.…”

Section: :2 • Park Et Almentioning

confidence: 99%

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Park

Lee

2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

In this work we present SUGO, a depth video-based system for translating sign language to text using a smartphone's front camera. While exploiting depth-only videos offer benefits such as being less privacy-invasive compared to using RGB videos, it introduces new challenges which include dealing with low video resolutions and the sensors' sensitiveness towards user motion. We overcome these challenges by diversifying our sign language video dataset to be robust to various usage scenarios via data augmentation and design a set of schemes to emphasize human gestures from the input images for effective sign detection. The inference engine of SUGO is based on a 3-dimensional convolutional neural network (3DCNN) to classify a sequence of video frames as a pre-trained word. Furthermore, the overall operations are designed to be light-weight so that sign language translation takes place in real-time using only the resources available on a smartphone, with no help from cloud servers nor external sensing components. Specifically, to train and test SUGO, we collect sign language data from 20 individuals for 50 Korean Sign Language words, summing up to a dataset of ~5,000 sign gestures and collect additional in-the-wild data to evaluate the performance of SUGO in real-world usage scenarios with different lighting conditions and daily activities. Comprehensively, our extensive evaluations show that SUGO can properly classify sign words with an accuracy of up to 91% and also suggest that the system is suitable (in terms of resource usage, latency, and environmental robustness) to enable a fully mobile solution for sign language translation.

show abstract

“…We developed a data collection application for the Tizen Platform Wearable 2.3.2. The acceleration data collection application was developed in our previous works [17], [18] and [6]. The sampling rate of the 3D accelerometer is 20 Hz.…”

Section: Smartwatch Frameworkmentioning

confidence: 99%

Long Short-Term Network Based Unobtrusive Perceived Workload Monitoring with Consumer Grade Smartwatches in the Wild

Ekiz,

Can,

Ersoy

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Continuous high perceived workload has a negative impact on the individual's wellbeing. Prior works focused on detecting the workload with medical-grade wearable systems in the restricted settings, and the effect of applying deep learning techniques for perceived workload detection in the wild settings is not investigated. We present an unobtrusive, comfortable, pervasive and affordable Long Short-Term Memory Network based continuous workload monitoring system based on a smartwatch application that monitors the perceived workload of individuals in the wild. We make use of modern consumer-grade smartwatches. We have recorded physiological data from daily life with perceived workload questionnaires from subjects in their real-life environments over a month. The model was trained and evaluated with the daily-life physiological data coming from different days which makes it robust to daily changes in the heart rate variability, that we use with accelerometer features to asses low and high workload. Our system has the capability of removing motion-related artifacts and detecting perceived workload by using traditional and deep classifiers. We discussed the problems related to in the wild applications with the consumer-grade smartwatches. We showed that Long Short-Term Memory Network outperforms traditional classifiers on discrimination of low and high workload with smartwatches in the wild.

show abstract

Sign sentence recognition with smart watches

Cited by 20 publications

References 5 publications

A Portable Sign Language Collection and Translation Platform with Smart Watches Using a BLSTM-Based Multi-Feature Framework

A Portable Sign Language Collection and Translation Platform with Smart Watches Using a BLSTM-Based Multi-Feature Framework

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

Long Short-Term Network Based Unobtrusive Perceived Workload Monitoring with Consumer Grade Smartwatches in the Wild

Contact Info

Product

Resources

About