Unsupervised classification of extreme facial events using active appearance models tracking for sign language videos

Antonakos, Epameinondas; Pitsikalis, Vassilis; Rodomagoulakis, Isidoros; Maragos, Petros

doi:10.1109/icip.2012.6467133

Cited by 9 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overall, different methods focus partially on some of the above aspects, and to the best of our knowledge, none of them shares all described issues. In [43], we introduced ESC. Herein, the approach is extensively presented with mature and updated material, including the application to multiple events and more experiments.…”

Section: Relative Literaturementioning

confidence: 99%

“…This spreading is controlled through a parameter that we call Subjective Perceived Threshold (SPThres). In our previous work [43], we formed clusters by selecting pairs of data points that have maximum distance until reaching each time the selected scalar SPThres value. In this work, the SPThres value is determined depending on the introduced facial event symmetry.…”

Section: Cluster Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Classification of extreme facial events in sign language videos

2014

Self Cite

View full text Add to dashboard Cite

We propose a new approach for Extreme States Classification (ESC) on feature spaces of facial cues in sign language (SL) videos. The method is built upon Active Appearance Model (AAM) face tracking and feature extraction of global and local AAMs. ESC is applied on various facial cues -as, for instance, pose rotations, head movements and eye blinking -leading to the detection of extreme states such as left/right, up/down and open/closed. Given the importance of such facial events in SL analysis, we apply ESC to detect visual events on SL videos, including both American (ASL) and Greek (GSL) corpora, yielding promising qualitative and quantitative results. Further, we show the potential of ESC for assistive annotation tools and demonstrate a link of the detections with indicative higher-level linguistic events. Given the lack of facial annotated data and the fact that manual annotations are highly time-consuming, ESC results indicate that the framework can have significant impact on SL processing and analysis.

show abstract

Section: Relative Literaturementioning

confidence: 99%

Section: Cluster Selectionmentioning

confidence: 99%

Classification of extreme facial events in sign language videos

2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…Analogous to phonemes in speech, sign languages can be broken down into cheremes, the smallest distinctive structural units [58]. Cheremes can be represented as motion primitives, a set of manual and non-manual motions 1 that are combined to represent all sign language utterances. Such phonetic representations are typically used by linguists for annotation [29,62] or in graphical based avatars for sign generation.…”

Section: Introductionmentioning

confidence: 99%

Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

Saunders

Camgöz

Bowden

2021

Preprint

View full text Add to dashboard Cite

It is common practice to represent spoken languages at their phonetic level. However, for sign languages, this implies breaking motion into its constituent motion primitives. Avatar based Sign Language Production (SLP) has traditionally done just this, building up animation from sequences of hand motions, shapes and facial expressions. However, more recent deep learning based solutions to SLP have tackled the problem using a single network that estimates the full skeletal structure.We propose splitting the SLP task into two distinct jointly-trained sub-tasks. The first translation sub-task translates from spoken language to a latent sign language representation, with gloss supervision. Subsequently, the animation sub-task aims to produce expressive sign language sequences that closely resemble the learnt spatiotemporal representation. Using a progressive transformer for the translation sub-task, we propose a novel Mixture of Motion Primitives (MOMP) architecture for sign language animation. A set of distinct motion primitives are learnt during training, that can be temporally combined at inference to animate continuous sign language sequences.We evaluate on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, presenting extensive ablation studies and showing that MOMP outperforms baselines in user evaluations. We achieve state-of-the-art back translation performance with an 11% improvement over competing results. Importantly, and for the first time, we showcase stronger performance for a full translation pipeline going from spoken language to sign, than from gloss to sign.1 Manual features are the hand shape and motion, whereas non-manuals are the mouthings and facial expressions 2 Glosses are a written representation of sign, defined as minimal lexical items.

show abstract

“…They also discover a complex interaction between head position and mouth shape. Antonakos et al [2], [3] propose a novel semi-supervised approach for Extreme States Classification (ESC) on feature spaces of facial cues in SL videos. Their method is built upon AAM face tracking and feature extraction of global and local AAMs and applied for detection of sign boundaries and alternative constructions.…”

Section: Mouth Non-manuals In Existing Aslr Systemsmentioning

confidence: 99%

A survey on mouth modeling and analysis for Sign Language recognition

Antonakos

Roussos

Zafeiriou

2015

2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)

Self Cite

View full text Add to dashboard Cite

Abstract-Around 70 million Deaf worldwide use Sign Languages (SLs) as their native languages. At the same time, they have limited reading/writing skills in the spoken language. This puts them at a severe disadvantage in many contexts, including education, work, usage of computers and the Internet. Automatic Sign Language Recognition (ASLR) can support the Deaf in many ways, e.g. by enabling the development of systems for Human-Computer Interaction in SL and translation between sign and spoken language. Research in ASLR usually revolves around automatic understanding of manual signs. Recently, ASLR research community has started to appreciate the importance of non-manuals, since they are related to the lexical meaning of a sign, the syntax and the prosody. Nonmanuals include body and head pose, movement of the eyebrows and the eyes, as well as blinks and squints. Arguably, the mouth is one of the most involved parts of the face in non-manuals. Mouth actions related to ASLR can be either mouthings, i.e. visual syllables with the mouth while signing, or non-verbal mouth gestures. Both are very important in ASLR. In this paper, we present the first survey on mouth non-manuals in ASLR. We start by showing why mouth motion is important in SL and the relevant techniques that exist within ASLR. Since limited research has been conducted regarding automatic analysis of mouth motion in the context of ALSR, we proceed by surveying relevant techniques from the areas of automatic mouth expression and visual speech recognition which can be applied to the task. Finally, we conclude by presenting the challenges and potentials of automatic analysis of mouth motion in the context of ASLR.

show abstract

Unsupervised classification of extreme facial events using active appearance models tracking for sign language videos

Cited by 9 publications

References 12 publications

Classification of extreme facial events in sign language videos

Classification of extreme facial events in sign language videos

Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

A survey on mouth modeling and analysis for Sign Language recognition

Contact Info

Product

Resources

About