MLSLT: Towards Multilingual Sign Language Translation

Aoxiong, Yin,; Zhao, Zhou; Jin, Weike; Zhang, Meng; Zeng, Xingshan; He, Xiaofei

doi:10.1109/cvpr52688.2022.00505

Cited by 25 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2022, the first SLT dataset containing parallel data in multiple sign languages was introduced [98]. The SP-10 dataset contains sign language data and parallel translations from ten sign languages.…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Machine translation from signed to spoken languages: state of the art and challenges

Coster

Shterionov

Herreweghe

et al. 2023

Univ Access Inf Soc

View full text Add to dashboard Cite

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity—the majority of scientific papers on sign language (SL) translation have been published in the past five years—research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

show abstract

Section: Datasetsmentioning

confidence: 99%

“…It was created from data collected in the SpreadTheSign research project [271]. Yin et al [98] show that multilingual training of SLT models can improve performance and allow for zero-shot translation.…”

Section: Datasetsmentioning

confidence: 99%

Machine translation from signed to spoken languages: state of the art and challenges

Coster

Shterionov

Herreweghe

et al. 2023

Univ Access Inf Soc

View full text Add to dashboard Cite

show abstract

“…The variations in signing grammar, vocabulary, and cultural influences pose challenges. Deep learning can adapt well but requires language-specific training data and careful consideration of cultural nuances [25][26][27].…”

Section: Slr For Other Languages (Beyond Asl)mentioning

confidence: 99%

Sign-to-Text Translation from Panamanian Sign Language to Spanish in Continuous Capture Mode with Deep Neural Networks

Teran-Quezada,

Lopez-Cabrera,

Rangel

et al. 2024

BDCC

View full text Add to dashboard Cite

Convolutional neural networks (CNN) have provided great advances for the task of sign language recognition (SLR). However, recurrent neural networks (RNN) in the form of long–short-term memory (LSTM) have become a means for providing solutions to problems involving sequential data. This research proposes the development of a sign language translation system that converts Panamanian Sign Language (PSL) signs into text in Spanish using an LSTM model that, among many things, makes it possible to work with non-static signs (as sequential data). The deep learning model presented focuses on action detection, in this case, the execution of the signs. This involves processing in a precise manner the frames in which a sign language gesture is made. The proposal is a holistic solution that considers, in addition to the seeking of the hands of the speaker, the face and pose determinants. These were added due to the fact that when communicating through sign languages, other visual characteristics matter beyond hand gestures. For the training of this system, a data set of 330 videos (of 30 frames each) for five possible classes (different signs considered) was created. The model was tested having an accuracy of 98.8%, making this a valuable base system for effective communication between PSL users and Spanish speakers. In conclusion, this work provides an improvement of the state of the art for PSL–Spanish translation by using the possibilities of translatable signs via deep learning.

show abstract

“…To address the weakly supervised problem in SLT, Song et al [50] proposed a parallel temporal encoder framework to extract both global and local information simultaneously. To address the problem of multilanguage SLT (MSLT), Yin et al [62] proposed a transformer model for translating different types of sign language into corresponding texts in an end-to-end way. To capture the nonlocal and global semantic information of the video and spoken language text, Guo et al [63] proposed a locality-aware transformer (LAT) framework for the task of SLT.…”

Section: Network Structure For Improving the Performance Of Translationmentioning

confidence: 99%

“…Spreadthesign-Ten (SP-10) [62] is a multilingual sign language dataset for SLR. The sign language videos and corresponding texts of SP-10 were collected from 10 different languages.…”

Section: Datasetsmentioning

confidence: 99%

Sign Language Translation: A Survey of Approaches and Techniques

Liang

Chai

2023

Electronics

View full text Add to dashboard Cite

Sign language is the main communication way for deaf and hard-of-hearing (i.e., DHH) people, which is unfamiliar to most non-deaf and hard-of-hearing (non-DHH) people. To break down the communication barriers between DHH and non-DHH people and to better promote communication among DHH individuals, we have summarized the research progress on sign language translation. We provide the necessary background on sign language translation and introduce its four subtasks (i.e., sign2gloss2text, sign2text, sign2(gloss+text), and gloss2text). We distill the basic mode of sign language translation (SLT) and introduce the transformer-based framework of SLT. We analyze the main challenges of SLT and propose possible directions for its development.

show abstract

MLSLT: Towards Multilingual Sign Language Translation

Cited by 25 publications

References 23 publications

Machine translation from signed to spoken languages: state of the art and challenges

Machine translation from signed to spoken languages: state of the art and challenges

Sign-to-Text Translation from Panamanian Sign Language to Spanish in Continuous Capture Mode with Deep Neural Networks

Sign Language Translation: A Survey of Approaches and Techniques

Contact Info

Product

Resources

About