A Kinect based sign language recognition system using spatio-temporal features

Memiş, Abbas; Albayrak, Songul

doi:10.1117/12.2051018

Cited by 15 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The study presented maximum accuracy for 15word recognition by 98 %. Memis and Albayrak [10] developed a system that recognized 111 TSL words in 3 different classes with the k-NN machine learning method employing scenes provided by the Kinect sensor. They tested the recognition system using the 3-fold method and were able to achieve 91.52 % recognition accuracy rate.…”

Section: Related Studiesmentioning

confidence: 99%

Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted Features

Karacı

Akyol

Turut

2021

Applied Computer Systems

View full text Add to dashboard Cite

In this study, a machine learning-based system, which recognises the Turkish sign language person-independent in real-time, was developed. A leap motion sensor was used to obtain raw data from individuals. Then, handcraft features were extracted by using Euclidean distance on the raw data. Handcraft features include finger-to-finger, finger -to-palm, finger -to-wrist bone, palm-to-palm and wrist-to-wrist distances. LR, k-NN, RF, DNN, ANN single classifiers were trained using the handcraft features. Cascade voting approach was applied with two-step voting. The first voting was applied for each classifier’s final prediction. Then, the second voting, which voted the prediction of all classifiers at the final decision stage, was applied to improve the performance of the proposed system. The proposed system was tested in real-time by an individual whose hand data were not involved in the training dataset. According to the results, the proposed system presents 100 % value of accuracy in the classification of one hand letters. Besides, the recognition accuracy ratio of the system is 100 % on the two hands letters, except “J” and “H” letters. The recognition accuracy rates were 80 % and 90 %, respectively for “J” and “H” letters. Overall, the cascade voting approach presented a high average classification performance with 98.97 % value of accuracy. The proposed system enables Turkish sign language recognition with high accuracy rates in real time.

show abstract

Section: Related Studiesmentioning

confidence: 99%

Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted Features

Karacı

Akyol

Turut

2021

Applied Computer Systems

View full text Add to dashboard Cite

show abstract

“…Early studies utilized handcrafted features, such as scale invariant feature transform (SIFT) [18], [30], histogram of gradient (HOG) [16], [28], [33]. After feature extraction, features are fed into a classifier such as support vector ma-chine (SVM) [18], [30], K-nearest neighbour (K-NN) [19], or sequence models such as Hidden Markov Models (HMMs) [11], [16], [17], [26]. Also, some studies use dynamic time warping (DTW), a time series matching algorithm, for recognition [28], [33].…”

Section: Related Workmentioning

confidence: 99%

“…In the field of TSL, some early domain specific research are conducted for special purposes, e.g., [22]- [24] aims to assist TSL education, [25] implements human computer interaction systems in health and finance domains. Due to the absence of publicly available large-scale TSL datasets, researchers have to create their own small scale datasets for the development of special purpose SLR systems [19], [20], [25]. There is a need for a new publicly accessible large-scale TSL dataset to provide the ground for various researches in this domain, especially using the recent deep learning techniques.…”

Section: Introductionmentioning

confidence: 99%

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

2020

View full text Add to dashboard Cite

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains color image (RGB), depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used Convolutional Neural Networks (CNNs) to extract features, unidirectional and bidirectional Long Short-Term Memory (LSTM) models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

show abstract

“…82 The spatio-temproal features are obtained by applying a 2D discrete cosine transform (DCT) to the accumulated Kinect color and depth images.…”

Section: Speech and Sign Language Recognitionmentioning

confidence: 99%

A Survey of Applications and Human Motion Recognition with Microsoft Kinect

Lun

Zhao

2015

Int. J. Patt. Recogn. Artif. Intell.

281

109

View full text Add to dashboard Cite

Microsoft Kinect, a low-cost motion sensing device, enables users to interact with computers or game consoles naturally through gestures and spoken commands without any other peripheral equipment. As such, it has commanded intense interests in research and development on the Kinect technology. In this paper, we present, a comprehensive survey on Kinect applications, and the latest research and development on motion recognition using data captured by the Kinect sensor. On the applications front, we review the applications of the Kinect technology in a variety of areas, including healthcare, education and performing arts, robotics, sign language recognition, retail services, workplace safety training, as well as 3D reconstructions. On the technology front, we provide an overview of the main features of both versions of the Kinect sensor together with the depth sensing technologies used, and review literatures on human motion recognition techniques used in Kinect applications. We provide a classi¯cation of motion recognition techniques to highlight the di®erent approaches used in human motion recognition. Furthermore, we compile a list of publicly available Kinect datasets. These datasets are valuable resources for researchers to investigate better methods for human motion recognition and lowerlevel computer vision tasks such as segmentation, object detection and human pose estimation.

show abstract

A Kinect based sign language recognition system using spatio-temporal features

Cited by 15 publications

References 6 publications

Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted Features

Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted Features

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

A Survey of Applications and Human Motion Recognition with Microsoft Kinect

Contact Info

Product

Resources

About