43. Transcription

Frishberg, Nancy; Hoiting, Nini; Slobin, Dan I.

doi:10.1515/9783110261325.1045

Cited by 37 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method is similar to written language MT: they both translate from text to text. However, glosses do not convey the full meaning of what is being signed [20,21] and they are often influenced by written language [20,22]. Hence, end-to-end SLT (from video rather than glosses), is required to fully extract the meaning of the sign language utterances.…”

Section: Sign Language Machine Translationmentioning

confidence: 99%

“…However, translating from sign language glosses to written language text is not desirable. Glosses do not convey the full meaning of signing [20,22] and may present an information bottleneck [14]. Nevertheless, they can still provide useful information to bootstrap the learning process of a sign language translation system.…”

Section: Sign Language Machine Translationmentioning

confidence: 99%

See 1 more Smart Citation

Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation

Coster

Dambre

2022

Information

View full text Add to dashboard Cite

We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.

show abstract

Section: Sign Language Machine Translationmentioning

confidence: 99%

Section: Sign Language Machine Translationmentioning

confidence: 99%

Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation

Coster

Dambre

2022

Information

View full text Add to dashboard Cite

show abstract

“…Unlike many spoken languages, sign languages do not have a standardized written form. Several notation systems do exist, but none of them are generally accepted as a standard [30]. The earliest notation system was proposed in the 1960s by Stokoe: the Stokoe notation [65].…”

Section: Notation Systems For Sign Languagesmentioning

confidence: 99%

“…Examples of glosses are: "CAR", "BRIDGE", but also "CAR-CROSSES-BRIDGE". Glosses do not accurately represent the meaning of signs in all cases and glossing has several limitations and problems [29]. They are inherently sequential, while signs often exhibit simultaneity [75] 5 .…”

Section: Notation Systems For Sign Languagesmentioning

confidence: 99%

“…They are inherently sequential, while signs often exhibit simultaneity [75] 5 . Furthermore, as glosses are based on spoken languages, there may be an implicit influence of the spoken language projected onto the sign language [74,29]. Finally, there is no universal standard on how glosses should be constructed, leading to differences between corpora of different sign languages, or even between several sign language annotators working on the same corpus.…”

Section: Notation Systems For Sign Languagesmentioning

confidence: 99%

See 1 more Smart Citation

Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

Coster¹,

Shterionov²,

Herreweghe³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic translation from signed to spoken languages is an interdisciplinary research domain, lying on the intersection of computer vision, machine translation and linguistics. Nevertheless, research in this domain is performed mostly by computer scientists in isolation. As the domain is becoming increasingly popular -the majority of scientific papers on the topic of sign language translation have been published in the past three years -we provide an overview of the state of the art as well as some required background in the different related disciplines. We give a high-level introduction to sign language linguistics and machine translation to illustrate the requirements of automatic sign language translation. We present a systematic literature review to illustrate the state of the art in the domain and then, harking back to the requirements, lay out several challenges for future research. We find that significant advances have been made on the shoulders of spoken language machine translation research. However, current approaches are often not linguistically motivated or are not adapted to the different input modality of sign languages. We explore challenges related to the representation of sign language data, the collection of datasets, the need for interdisciplinary research and requirements for moving beyond research, towards applications. Based on our findings, we advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages. Furthermore, the inclusion of deaf and hearing end users of sign language translation applications in use case identification, data collection and evaluation is of the utmost importance in the creation of useful sign language translation models. We recommend iterative, human-in-the-loop, design and development of sign language translation models.

show abstract