Fingerspelling Recognition through Classification of Letter-to-Letter Transitions

Ricco, Susanna; Tomasi, Carlo

doi:10.1007/978-3-642-12297-2_21

Cited by 24 publications

(22 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the represented mapping, our objective function predicts an entry rate of 43.9 WPM which is surprisingly close to the empirically observed rate of 40-45 WPM for experienced practitioners [30].…”

Section: Design Casessupporting

confidence: 78%

Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

Sridhar

Feit²,

Theobalt

et al. 2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Figure 1. We investigate the dexterity of using multiple fingers for mid-air input. The paper reports performance and individuation characteristics of fingers and deploys them to the design of a mid-air text entry method using multi-objective optimization. Here we show an example of the word 'hand' being typed using one of our automatically obtained designs. ABSTRACTThis paper investigates an emerging input method enabled by progress in hand tracking: input by free motion of fingers. The method is expressive, potentially fast, and usable across many settings as it does not insist on physical contact or visual feedback. Our goal is to inform the design of high-performance input methods by providing detailed analysis of the performance and anatomical characteristics of finger motion. We conducted an experiment using a commercially available sensor to report on the speed, accuracy, individuation, movement ranges, and individual differences of each finger. Findings show differences of up to 50% in movement times and provide indices quantifying the individuation of single fingers. We apply our findings to text entry by computational optimization of multi-finger gestures in mid-air. To this end, we define a novel objective function that considers performance, anatomical factors, and learnability. First investigations of one optimization case show entry rates of 22 words per minute (WPM). We conclude with a critical discussion of the limitations posed by human factors and performance characteristics of existing markerless hand trackers.

show abstract

Section: Design Casessupporting

confidence: 78%

Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

Sridhar

Feit²,

Theobalt

et al. 2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…The problem can be approached similarly to speech recognition, with letters being the analogues of words or phones, and most work thus far has indeed used HMM-based approaches with HMMs representing letters (e.g., [11,4]) or letter-to-letter transitions [12]. Most prior work has limited the vocabulary to 20-100 words, in which case it is common to obtain letter error rates of 10% or less.…”

Section: Introductionmentioning

confidence: 99%

American sign language fingerspelling recognition with phonological feature-based tandem models

Kim

Livescu

Shakhnarovich

2012

2012 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

We study the recognition of fingerspelling sequences in American Sign Language from video using tandem-style models, in which the outputs of multilayer perceptron (MLP) classifiers are used as observations in a hidden Markov model (HMM)-based recognizer. We compare a baseline HMMbased recognizer, a tandem recognizer using MLP letter classifiers, and a tandem recognizer using MLP classifiers of phonological features. We present experiments on a database of fingerspelling videos. We find that the tandem approaches outperform an HMM-based baseline, and that phonological feature-based tandem models outperform letter-based tandem models.

show abstract

“…A subset of ASL recognition work has focused on fingerspelling and/or handshape classification [4,27,24] and fingerspelling sequence recognition [14,20,26], where letter error rates of 10% or less have been achieved when the 1 A good deal of prior work on sign language recognition has also used other instrumentation, such as specialized gloves or depth maps [17,16,21,39,13]. These interfaces are sometimes feasible, but video remains more practical in many settings, and we restrict our discussion to video here.…”

Section: Related Workmentioning

confidence: 99%

“…Most prior work on fingerspelling recognition has assumed a closed vocabulary of fingerspelled words, often limited to 20-100 words, typically using hidden Markov models (HMMs) representing letters or letter-to-letter transitions [14,20,26]. In such settings it is common to obtain letter error rates (Levenshtein distances between hypothesized and true letter sequences, as a proportion of the number of true letters) of 10% or less.…”

Section: Introductionmentioning

confidence: 99%

Fingerspelling Recognition with Semi-Markov Conditional Random Fields

Kim

Shakhnarovich

Livescu

2013

2013 IEEE International Conference on Computer Vision

View full text Add to dashboard Cite

Recognition of gesture sequences is in general a very difficult problem, but in certain domains the difficulty may be mitigated by exploiting the domain's "grammar". One such grammatically constrained gesture sequence domain is sign language. In this paper we investigate the case of fingerspelling recognition, which can be very challenging due to the quick, small motions of the fingers. Most prior work on this task has assumed a closed vocabulary of fingerspelled words; here we study the more natural open-vocabulary case, where the only domain knowledge is the possible fingerspelled letters and statistics of their sequences. We develop a semi-Markov conditional model approach, where feature functions are defined over segments of video and their corresponding letter labels. We use classifiers of letters and linguistic handshape features, along with expected motion profiles, to define segmental feature functions. This approach improves letter error rate (Levenshtein distance between hypothesized and correct letter sequences) from 16.3% using a hidden Markov model baseline to 11.6% using the proposed semi-Markov model.

show abstract

Fingerspelling Recognition through Classification of Letter-to-Letter Transitions

Cited by 24 publications

References 23 publications

Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

American sign language fingerspelling recognition with phonological feature-based tandem models

Fingerspelling Recognition with Semi-Markov Conditional Random Fields

Contact Info

Product

Resources

About