Quality-Aware Decoding for Neural Machine Translation

Fernandes, Patrick; Farinhas, António; Rei, Ricardo; Souza, José de; Perez, Ogayo,; Neubig, Graham; Martins, André F. T.

doi:10.18653/v1/2022.naacl-main.100

Cited by 29 publications

(35 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When using LLM-SCQ, the performance of MAPS is on par with 5-Shot (MAPS LLM-SCQ ≈ 5-Shot); when using COMET-QE, MAPS consistently outperforms 5-Shot (MAPS COMET-QE > 5-Shot). More importantly, MAPS shows higher upper bounds for selection than Rerank (MAPS COMET > Rerank COMET ), implying that superior knowledge selection methods like a better QE model (Rei et al, 2022b), AutoMQM (Fernandes et al, 2023) or ranking strategy (Fernandes et al, 2022) can further improve MAPS.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Exploring Human-Like Translation Strategy with Large Language Models

He,

Liang,

Jiao

et al. 2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Large language models (LLMs) have demonstrated impressive capabilities in general scenarios, exhibiting a level of aptitude that approaches, in some aspects even surpasses, human-level intelligence. Among their numerous skills, the translation abilities of LLMs have received considerable attention. Compared to typical machine translation that focuses solely on source-to-target mapping, LLM-based translation can potentially mimic the human translation process, which might take preparatory steps to ensure high-quality translation. This work explores this possibility by proposing the MAPS framework, which stands for Multi-Aspect Prompting and Selection. Specifically, we enable LLMs first to analyze the given source sentence and induce three aspects of translation-related knowledge (keywords, topics, and relevant demonstrations) to guide the final translation process. Moreover, we employ a selection mechanism based on quality estimation to filter out noisy and unhelpful knowledge. Both automatic (3 LLMs × 11 directions × 2 automatic metrics) and human evaluation (preference study and MQM) demonstrate the effectiveness of MAPS. Further analysis shows that by mimicking the human translation process, MAPS reduces various translation errors such as hallucination, ambiguity, mistranslation, awkward style, untranslated text, and omission. Source code is available at https://github.com/zwhe99/MAPS-mt.

show abstract

Section: Resultsmentioning

confidence: 99%

“…The best candidate is selected through QE. It can be considered as a pure reranking method without any guidance from extracted knowledge (Fernandes et al, 2022).…”

Section: Methodsmentioning

confidence: 99%

Exploring Human-Like Translation Strategy with Large Language Models

He,

Liang,

Jiao

et al. 2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Results Interpretation: The Table enlists quality metrics of each algorithm over the given datasets. The Decision Tree-based Explanation is the most accurate method among all the methods evaluated, and outperforms all other methods in all datasets [12]. This seems to be a case where model tree-based approach for decision trees has more accurate understanding of the model predictions than model agnostic approach like LIME and SHAP.…”

Section: Evaluation Metricsmentioning

confidence: 86%

“…The writers applied Convolutional Neural Networks (CNNs) and Gradient Boosting Machines (GBMs) for image classification, finance, and healthcare processes respectively. • Explanation Algorithms: LIME, SHAP, Anchor, and Decision tree-based explanation is the method we utilized to make explanations for each of the black-box models predictions [12]. The interpretation of each explanation algorithm was determined by its capacity to give interpretative insights into predictions.…”

Section: Evaluation Metricsmentioning

confidence: 99%

Explainable AI: Bridging the Gap between Machine Learning Models and Human Understanding

2024

Journal of Informatics Education and Research

View full text Add to dashboard Cite

Explainable AI (XAI) is one of the key game-changing features in machine learning models, which contribute to making them more transparent, regulated and usable in different applications. In (the) investigation of this paper, we consider the four rows of explanation methods-LIME, SHAP, Anchor, and Decision Tree-based Explanation-in disentangling the decision-making process of the black box models within different fields. In our experiments, we use datasets that cover different domains, for example, health, finance and image classification, and compare the accuracy, fidelity, coverage, precision and human satisfaction of each explanation method. Our work shows the rule trees approach called (Decision Tree-based explanation) is mostly superior in comparison to other non-model-specific methods of explanation performing higher accuracy, fidelity, coverage and precision regardless the classifier. In addition to this, the respondents who answered the qualitative evaluation indicated that they were very content with the decision tree-based explanations and that these types of explanations are very easy and understandable. Furthermore, most of the respondents famous that these sorts of clarifications are more instinctive and significant. The over discoveries stretch on the utilize of interpretable AI strategies for facilitating the hole between machine learning models and human understanding and thus advancing straightforwardness and responsibility in AI-driven decision-making.

show abstract

“…Our proposal RL4F is orthogonal to RLHF; in fact we use an RLHF fine-tuned checkpoint in our experiments. For further discussion, please refer to Fernandes et al (2023) who catalogue different approaches on integrating natural language feedback to textual generations.…”

Section: How Is Feedback Used?mentioning

confidence: 99%

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Akyürek¹,

Ekin²,

Kalyan³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrievalaugmented or prompting-based critique generators. 1

show abstract

Quality-Aware Decoding for Neural Machine Translation

Cited by 29 publications

References 36 publications

Exploring Human-Like Translation Strategy with Large Language Models

Exploring Human-Like Translation Strategy with Large Language Models

Explainable AI: Bridging the Gap between Machine Learning Models and Human Understanding

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Contact Info

Product

Resources

About