Eye tracking as an MT evaluation technique

Abstract

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc for some time now. Its application as a technique for automatically measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semiand/or automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. 10 native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded noninvasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the "bad" sentences, while average fixation duration and pupil dilations were not found to be substantially different between output rated as good or bad. Comparisons between BLEU scores and eye gaze data were also made and found to correlate well with gaze time and fixation count, and to a lesser extent with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as an automatic MT Evaluation technique.

Measuring cognitive load in the presence of educational video: Towards a multimodal methodology

“…382). Therefore, longer mean fixation durations within the same activity could reflect increased cognitive processing and cognitive load (Kruger et al, 2015), a finding consistently reported in other studies (e.g., Doherty & O'Brien, 2014;Doherty, O'Brien, & Carl, 2010). In this regard, however, it is important to consider that activities like reading tend to elicit more, shorter fixations than scene perception, which elicits fewer, longer fixations (cf.…”

Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators

“…Cognitive processing and effort can be studied via gaze data, building on the eye-mind hypothesis by Just and Carpenter (1980). An increased number of fixations (Doherty, O'Brien, et al 2010) or higher average fixation durations (Carl, Dragsted, Elming, et al 2011) have been used as indicators of increased cognitive processing. O'Brien (2007) compared human translation with post-editing and translation memory (TM) matches and found postediting to be less cognitively demanding than human translation.…”

“…O'Brien (2007) compared human translation with post-editing and translation memory (TM) matches and found postediting to be less cognitively demanding than human translation. Doherty, O'Brien, et al (2010) used eye-tracking to identify MT comprehensibility and discovered that the total gaze time and number of fixations correlate well with MT quality. When considering the difference between source text processing and target text processing, findings become more complicated.…”

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

“…The current study builds upon previous work conducted by the researchers in which the usefulness of eye tracking as a semi-automatic method for the evaluation of MT output was investigated (Doherty and O'Brien, 2009;Doherty et al, 2010, Doherty andO'Brien, 2012), in addition to the effects of controlled authoring 1 on the readability and comprehension of MT output (O'Brien, 2010;Doherty, 2012). Related work includes that of Tomita (1992) and Tomita et al (1993), who examined the comprehension of MT output, and of Fuji (1999), who, in addition to this measure, included informativeness and fluency.…”