Testing Machine Translation via Referential Transparency

He, Pinjia; Meister, Clara; Su, Zhendong

doi:10.1109/icse43902.2021.00047

Cited by 42 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Behavioral Testing for MT A number of previous works Gupta et al, 2020;Sun et al, 2020;Wang et al, 2021;He et al, 2021) have tried to construct tests for eliciting errors in NMT systems' behavior. We present a thorough comparison of SALTED against these works along five dimensions in appendix G.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SALTED: A Framework for SAlient Long-Tail Translation Error Detection

Raunak¹,

Post²,

Menezes³

2022

Preprint

View full text Add to dashboard Cite

Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, dropped content and hallucinations. These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems. Consequently, it is important to have visibility into these problems during model development. Towards this direction, we introduce SALTED, a specifications-based framework for behavioral testing of MT models that provides fine-grained views of salient long-tail errors, permitting trustworthy visibility into previously invisible problems. At the core of our approach is the development of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model finetuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.2. We propose an iterative, specifications-based process for obtaining reliable measurements through high-precision detectors and demonstrate their utility on seven MT error classes across both research and commercial systems.3. We demonstrate that detectors are amenable to multiple applications in MT, including higher-recall training data filtering, systemcomparisons, metamorphic testing and fixing errors through fine-tuning on synthetic data. The SALTED ApproachBehavioral Testing (Beizer, 1995) concerns itself with testing the input-output behavior of systems, without leveraging any knowledge about the system's internal structure. For behavioral testing

show abstract

Section: Related Workmentioning

confidence: 99%

“…22: A comparison of existing Behavioral Testing Methods for NMT along five dimensions. The compared methods are: SIT , PatInv (Gupta et al, 2020), TransRepair (Sun et al, 2020) and RTI (He et al, 2021).…”

mentioning

confidence: 99%

SALTED: A Framework for SAlient Long-Tail Translation Error Detection

Raunak¹,

Post²,

Menezes³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These two tasks can be considered as translation tasks in general, and there is a need to quantify the uncertainty of the translated results, i.e., the model should not produce incorrect translation results if the output is uncertain. Indeed, numerous works have proposed techniques for testing such machine translation tasks for natural language processing [23,24,56], but these techniques focus exclusively on the erroneous in the translated sentences, rather than on their uncertainty. Additionally, the methods are limited to natural language processing, but our approach is suitable to code representation learning.…”

Section: Discussionmentioning

confidence: 99%

Energy-bounded Learning for Robust Models of Code

Bui¹,

Yu²

2021

Preprint

View full text Add to dashboard Cite

In programming, learning code representations has a variety of applications, including code classification, code search, comment generation, bug prediction, and so on. Various representations of code in terms of tokens, syntax trees, dependency graphs, code navigation paths, or a combination of their variants have been proposed, however, existing vanilla learning techniques have a major limitation in robustness, i.e., it is easy for the models to make incorrect predictions when the inputs are altered in a subtle way. To enhance the robustness, existing approaches focus on recognizing adversarial samples rather than on the valid samples that fall outside a given distribution, which we refer to as out-ofdistribution (OOD) samples. Recognizing such OOD samples is the novel problem investigated in this paper. To this end, we propose to first augment the in-distribution datasets with out-of-distribution samples such that, when trained together, they will enhance the model's robustness. We propose the use of an energy-bounded learning objective function to assign a higher score to in-distribution samples and a lower score to out-of-distribution samples in order to incorporate such out-of-distribution samples into the training process of source code models. In terms of OOD detection and adversarial samples detection, our evaluation results demonstrate a greater robustness for existing source code models to become more accurate at recognizing OOD data while being more resistant to adversarial attacks at the same time. Furthermore, the proposed energy-bounded score outperforms all existing OOD detection scores by a large margin, including the softmax confidence score, the Mahalanobis score, and ODIN.

show abstract

“…Word-level techniques [62,63] are based on word substitution, usually using synonyms sets or Pre-trained Language Models (PLMs). Sentence-level techniques [26] change the whole structure of the sentences either by adding a sentence to the original texts or transforming the entire texts into another semantically similar format. Those combining different levels of techniques [42] can be categorized into multi-level techniques.…”

Section: Preliminaries 21 Testing Techniques For Nlp Softwarementioning

confidence: 99%

AEON: A Method for Automatic Evaluation of NLP Test Cases

Huang¹,

Zhang²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Due to the labor-intensive nature of manual test oracle construction, various automated testing techniques have been proposed to enhance the reliability of Natural Language Processing (NLP) software. In theory, these techniques mutate an existing test case (e.g., a sentence with its label) and assume the generated one preserves an equivalent or similar semantic meaning and thus, the same label. However, in practice, many of the generated test cases fail to preserve similar semantic meaning and are unnatural (e.g., grammar errors), which leads to a high false alarm rate and unnatural test cases. Our evaluation study finds that 44% of the test cases generated by the state-of-the-art (SOTA) approaches are false alarms. These test cases require extensive manual checking effort, and instead of improving NLP software, they can even degrade NLP software when utilized in model training. To address this problem, we propose AEON for Automatic Evaluation Of NLP test cases. For each generated test case, it outputs scores based on semantic similarity and language naturalness. We employ AEON to evaluate test cases generated by four popular testing techniques on five datasets across three typical NLP tasks. The results show that AEON aligns the best with human judgment. In particular, AEON achieves the best average precision in detecting semantic inconsistent test cases, outperforming the best baseline metric by 10%. In addition, AEON also has the highest average precision of finding unnatural test cases, surpassing the baselines by more than 15%. Moreover, model training with test cases prioritized by AEON leads to models that are more accurate and robust, demonstrating AEON's potential in improving NLP software. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

Testing Machine Translation via Referential Transparency

Cited by 42 publications

References 44 publications

SALTED: A Framework for SAlient Long-Tail Translation Error Detection

SALTED: A Framework for SAlient Long-Tail Translation Error Detection

Energy-bounded Learning for Robust Models of Code

AEON: A Method for Automatic Evaluation of NLP Test Cases

Contact Info

Product

Resources

About