Dynamic Affine-Invariant Shape-Appearance Handshape Features and Classification in Sign Language Videos

Roussos, Anastasios; Theodorakis, Stavros; Pitsikalis, Vassilis; Maragos, Petros

doi:10.1007/978-3-319-57021-1_8

Cited by 7 publications

(8 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deformable masks provided by active appearance models (AAMs) [8] can successfully help with detecting and tracking several types of informative events of a sign sequence, e.g., eye blinking, as done in [9]. AAMs [10] have also significantly boosted the performance of handshape recognition in sign language videos.…”

Section: Related Workmentioning

confidence: 99%

Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Kratimenos

Pavlakos²,

Maragos

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Independent Sign Language Recognition is a complex visual recognition problem that combines several challenging tasks of Computer Vision due to the necessity to exploit and fuse information from hand gestures, body features and facial expressions. While many state-of-the-art works have managed to deeply elaborate on these features independently, to the best of our knowledge, no work has adequately combined all three information channels to efficiently recognize Sign Language. In this work, we employ SMPL-X, a contemporary parametric model that enables joint extraction of 3D body shape, face and hands information from a single image. We use this holistic 3D reconstruction for SLR, demonstrating that it leads to higher accuracy than recognition from raw RGB images and their optical flow fed into the state-of-the-art I3D-type network for 3D action recognition and from 2D Openpose skeletons fed into a Recurrent Neural Network. Finally, a set of experiments on the body, face and hand features showed that neglecting any of these, significantly reduces the classification accuracy, proving the importance of jointly modeling body shape, facial expression and hand pose for Sign Language Recognition.

show abstract

Section: Related Workmentioning

confidence: 99%

Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Kratimenos

Pavlakos²,

Maragos

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, in the scope of this work, hand shape recognition is seen as a classification task of a specific number of defined hand shapes. Known approaches fall into three categories: (i) template matching against a large data set of often synthetic gallery images [25] or contour shapes [1,3]; (ii) generative model fitting approaches [35,10,28]; and (iii) discriminative modelling approaches such as Cooper et al [6]. Cooper uses random forests trained on HOG features to distinguish 12 hand shapes, each trained on 1000 training samples.…”

Section: State-of-the-artmentioning

confidence: 99%

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Koller

Ney

Bowden

2016

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

223

150

View full text Add to dashboard Cite

This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.

show abstract

“…Three papers proposed novel methods within the area of sign language recognition (Cooper et al, 2012;Nayak et al, 2012;Roussos et al, 2013). (Cooper et al, 2012) describe a method for sign language recognition using linguistic subunits that are learned automatically by the system.…”

Section: Summary Of Special Topic Papers Not Related To the Challengesmentioning

confidence: 99%

“…Another benefit is that the method identifies the aspects of a sign that are least affected by movement epenthesis, i.e., by signs immediately preceding or following the sign in question. (Roussos et al, 2013) present a method for classifying handshapes for the purpose of sign language recognition. Cropped hand images are converted to a normalized representation called "shape-appearance images", based on a PCA decomposition of skin pixel colors.…”

Section: Summary Of Special Topic Papers Not Related To the Challengesmentioning

confidence: 99%

Challenges in Multi-modal Gesture Recognition

Escalera

Athitsos

Guyon

2017

The Springer Series on Challenges in Machine Learning

View full text Add to dashboard Cite

This paper surveys the state of the art on multimodal gesture recognition and introduces the JMLR special topic on gesture recognition 2011-2015. We began right at the start of the Kinect T M revolution when inexpensive infrared cameras providing image depth recordings became available. We published papers using this technology and other more conventional methods, including regular video cameras, to record data, thus providing a good overview of uses of machine learning and computer vision using multimodal data in this area of application. Notably, we organized a series of challenges and made available several datasets we recorded for that purpose, including tens of thousands of videos, which are available to conduct further research. We also overview recent state of the art works on gesture recognition based on a proposed taxonomy for gesture recognition, discussing challenges and future lines of research.

show abstract

Dynamic Affine-Invariant Shape-Appearance Handshape Features and Classification in Sign Language Videos

Cited by 7 publications

References 38 publications

Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

Challenges in Multi-modal Gesture Recognition

Contact Info

Product

Resources

About