Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

Abstract

This paper reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centred approach, we utilize the ISO/TR 16982 definition of usability -goal completion, satisfaction, effectiveness, and efficiency -and apply eye-tracking measures shown to be reliable indicators of cognitive effort, along with a post-task questionnaire. We investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, we found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw 2 machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.

Measuring cognitive load in the presence of educational video: Towards a multimodal methodology

“…382). Therefore, longer mean fixation durations within the same activity could reflect increased cognitive processing and cognitive load (Kruger et al, 2015), a finding consistently reported in other studies (e.g., Doherty & O'Brien, 2014;Doherty, O'Brien, & Carl, 2010). In this regard, however, it is important to consider that activities like reading tend to elicit more, shorter fixations than scene perception, which elicits fewer, longer fixations (cf.…”

Evaluating the Usability of a Controlled Language Authoring Assistant

“…In this study, we conducted a usability evaluation based on the ISO standard for human-computer interaction (ISO, 2010) and related studies (e.g., Doherty and O'Brien, 2013;Sauro and Lewis, 2012). To the best of our knowledge, this is the first attempt to evaluate the usability of a CL authoring assistant intended for improving MT performance.…”

How do Humans Evaluate Machine Translation

“…They found that longer gaze time and and a higher number of fixations correlate with high number of errors in the MT output. Doherty and O'Brien (2014) used eyetracking to evaluate the quality of raw machine translation output in terms of its usability by an end user. They concluded that eye-tracking correlates well with the other measures which they used for their study.…”