Real-Time Sign Language Detection Using Human Pose Estimation

Moryossef, Amit; Tsochantaridis, Ioannis; Aharoni, Roee; Ebling, Sarah; Narayanan, Srini

doi:10.1007/978-3-030-66096-3_17

Cited by 57 publications

(45 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This approach is based on location signing assumption, which our investigation has shown to be a viable cue and following the practice among TFS signers. The underlying spirit of a signing-location-based approach may be different from a concept of optical flow [6,13,14], but some implementation of optical flow estimation may arguably be close to D2M or other alternatives explored here. (An investigation of implementations of optical flow estimation is beyond the scope of this article.…”

Section: Discussionmentioning

confidence: 99%

“…However, a transition-based approach is susceptible to situations, such as unintended hand postures or rapid signing. Being aware of the risk of transcribing unintended hand postures during a non-signing session, some [14] partially address the issue using a full video-sequence analytic system, employing state-of-the-arts in deep computer vision and sequence modeling to distinguish between signing and non-signing sessions. Our ATFS is based on signing location, which is a convenient choice for that it is our cue for alphabet separation.…”

Section: Discussionmentioning

confidence: 99%

“…Therefore, we may refer that TFS has 27 multi-posture signings later on). The fact that TFS strongly relies on a multi-posture scheme makes a transcription task ambiguous, e.g., a sequence of TFS signs: 4, 7, 8, and 14 can be transcribed as [4,7,8,14], which all are treated as single postures, resulting "ตหวน" in Thai script, or transcribed as [4,7,8,14], whose first two signs are treated as a multi-posture signing, resulting "ทวน" in Thai script. Our work proposes to address this through a sequence model.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Looking beyond transcription, Moryossef et al [14] have their attention on distinguishing signing and non-signing sessions from a video performing German sign language. They employ part affinity field [15] to provide human pose estimation of each image frame.…”

Section: Automatic Thai Finger Spelling Transcriptionmentioning

confidence: 99%

“…Therefore, the transition approach is likely to be computationally cheaper than an image-feature-based approach, but this comes with a risk for unintended hand postures during non-signing sessions or fast signing during a signing session. To partially address it, Moryossef et al [14] propose a system utilizing part affinity field [15], optical flow, and LSTM dedicated to identifying signing and non-signing sessions.…”

Section: Automatic Thai Finger Spelling Transcriptionmentioning

confidence: 99%

See 4 more Smart Citations

Automatic Thai Finger Spelling Transcription

Nakjai

Katanyukul

2021

Walailak J Sci & Tech

View full text Add to dashboard Cite

This article explores a transcription of a video recording Thai Finger Spelling (TFS)—a specific signing mode used in Thai sign language—to a corresponding Thai word. TFS copes with 42 Thai alphabets and 20 vowels using multiple and complex schemes. This leads to many technical challenges uncommon in spelling schemes of other sign languages. Our proposed system, Automatic Thai Finger Spelling Transcription (ATFS), processes a signing video in 3 stages: ALS marking video frames to easily remove any non-signing frame as well as conveniently group frames associating to the same alphabet, SR classifying a signing image frame to a sign label (or its equivalence), and SSR transcribing a series of signs into alphabets. ALS utilizes the TFS practice of signing different alphabets at different locations. SR and SSC employ well-adopted spatial and sequential models. Our ATFS has been found to achieve Alphabet Error Rate (AER) 0.256 (c.f. 0.63 of the baseline method). In addition to ATFS, our findings have disclosed a benefit of coupling image classification and sequence modeling stages by using a feature or penultimate vector for label representation rather than a definitive label or one-hot coding. Our results also assert the necessity of a smoothening mechanism in ALS and reveal a benefit of our proposed WFS, which could lead to over 15.88 % improvement. For TFS transcription, our work emphasizes the utilization of signing location in the identification of different alphabets. This is contrary to a common belief of exploiting signing time duration, which are shown to be ineffective by our data. HIGHLIGHTS Prototype of Thai finger spelling transcription (transcribing a signing video to alphabets) Utilization of signing location as cue for identification of different alphabets Disclosure of a benefit of coupling image classification and sequence modeling in signing transcription Examination of various frame smoothing techniques and their contributions to the overall transcription performance GRAPHICAL ABSTRACT

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Automatic Thai Finger Spelling Transcriptionmentioning

confidence: 99%

Section: Automatic Thai Finger Spelling Transcriptionmentioning

confidence: 99%

See 3 more Smart Citations

Automatic Thai Finger Spelling Transcription

Nakjai

Katanyukul

2021

Walailak J Sci & Tech

View full text Add to dashboard Cite

show abstract

SLRTP 2020: The Sign Language Recognition, Translation & Production Workshop

Camgöz

Varol

Albanie

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The objective of the "Sign Language Recognition, Translation & Production" (SLRTP 2020) Workshop was to bring together researchers who focus on the various aspects of sign language understanding using tools from computer vision and linguistics. The workshop sought to promote a greater linguistic and historical understanding of sign languages within the computer vision community, to foster new collaborations and to identify the most pressing challenges for the field going forwards. The workshop was held in conjunction with the European Conference on Computer Vision (ECCV), 2020.

show abstract