Guidance to Pre-tokeniztion for SacreBLEU: Meta-Evaluation in Korean

Abstract

SacreBLEU, by incorporating a text normalizing step in the pipeline, has been well-received as an automatic evaluation metric in recent years. With agglutinative languages such as Korean, however, the metric cannot provide a conceivable result without the help of customized pre-tokenization. In this regard, this paper endeavors to examine the influence of diversified pre-tokenization schemes –word, morpheme, character, and subword– on the aforementioned metric by performing a meta-evaluation with manually-constructed into-Korean human evaluation data. Our empirical study demonstrates that the correlation of SacreBLEU (to human judgment) fluctuates consistently by the token type. The reliability of the metric even deteriorates due to some tokenization, and MeCab is not an exception. Guiding through the proper usage of tokenizer for each metric, we stress the significance of a character level and the insignificance of a Jamo level in MT evaluation.

Ainu–Japanese Bi-directional Neural Machine Translation: A Step Towards Linguistic Preservation of Ainu, An Under-Resourced Indigenous Language in Japan

“…Achieving a SacreBLEU score of 32.90 in the task of translating from Japanese to Ainu highlights the high quality of translations despite the Ainu language's complex polysynthetic structure Ortega et al [2020]. Furthermore, the accomplishment of a SacreBLEU score of 29.91 in the bi-directional translation task between Japanese and Ainu attests to the capability of the neural MT framework to handle intricate and resource-scarce languages efficiently Kim and Kim [2022b].…”

“…The positive SacreBLEU scores garnered from our experiments highlight MT's potential utility in supporting language preservation and revitalization projects. The data indicates that even within the constraints of limited linguistic resources, MT models can achieve a degree of accuracy that renders them valuable tools for learners and scholars engaged with the Ainu language Kim and Kim [2022b].…”

“…To evaluate our machine translation system, we employed the SacreBLEU metric Kim and Kim [2022b], a robust and reliable standard for measuring translation quality. SacreBLEU's consistent approach to tokenization and detokenization across evaluations is particularly advantageous, ensuring that comparisons between different MT systems or iterations are fair and uniform.…”