Prototyping and Preliminary Evaluation of Sign Language Translation System in the Railway Domain

Abstract

Abstract. This paper presents the prototype and the preliminary evaluation of an automatic translation system developed in the LIS4ALL project. The system domain is the corpus of railway station announcements in Italian. The output of the system is a 3D animated avatar that signs announcements in Italian Sign Language. The preliminary evaluation, which measures the accuracy of the translations at the sentence level, relies through the BLEU-RAC4 metric, a variant of the traditional BLEU metric used to evaluate Machine Translation, specifically designed for sentence level evaluation. The aim of the evaluation is to compare the LIS4ALL translation outputs with the human counterparts.

Sign Language Translation in a Healthcare Setting

“…In future work, we intend to evaluate the system more thoroughly and to employ the modular approach motivated here to develop text-to-sign translation systems for different domains, e.g., for announcements at airports or railway stations, a use case which has already been explored to some extent for other sign languages [2,8]. In addition, we also intend to improve the avatar visualisation.…”

“…In particular, automating the process of mapping input sentences to the corresponding NGT glosses using machine learning techniques would not have been feasible within a short time-frame, and would, even in the somewhat longer 117 term, most likely result in an unacceptably low accuracy rate for use in a healthcare setting. 2 We therefore mainly focused on automating the phonetic encoding step, something that significantly reduces the manual labor needed in the overall translation pipeline. Automating the mapping from glosses to phonetic representations has not been done in previous work on NGT [23] and, to the best of our knowledge, not in work on other sign languages either.…”

“…Phonetic approach: text =⇒ phonetic rep. =⇒ animation This approach has been taken in work based on SiGML and JASigning (see, e.g., [30,18,23,2,8,6]). Unlike in the gloss approach, applying machine learning techniques to carry out the first step, from text to phonetic representations, is not feasible because it would require the availability of large parallel corpora of texts and the corresponding phonetic sign representations, which do not exist and would be very costly to create.…”

Translating Italian to LIS in the Rail Stations

“…Finally, we intend to evaluate the quality of our translator by using both task-based human evaluation as well as metric-based automatic evaluation (Battaglino et al, 2015).…”