Automated Testing for Machine Translation via Constituency Invariance

Towards General Error Diagnosis via Behavioral Testing in Machine Translation

“…To avoid this challenge, existing work attempts to judge the translation quality of such test case without its reference translation, yet they can only diagnose translation errors related to specific types of edits that target certain capabilities. For example, ; Sun et al (2020); Gupta et al (2020); Ji et al (2021) only diagnose translation errors on test cases with the editing of a single noun or adjective word, and ; ; Raunak et al (2022) can only diagnose incorrect translation of noun phrases, quantities or currency units that is related to the edits on them.…”

“…In other words, the recall of translation errors identified by these approaches is low. On the other hand, Sun et al, 2020;Gupta et al, 2020;Ji et al, 2021) first edit a single noun or adjective in x to create a series of similar sentences as test cases, then denote M as passing the behavioral testing if its translations of these sentences have similar syntactic structures, based on an assumption that the translations of similar sentences should be analogous. The reason why they only modify a single noun or adjective is to avoid largely shifting the structure of x, yet still limits the types of capability they can test as well as translation errors they can find.…”