How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?

Amrhein, Chantal; Sennrich, Rico

doi:10.18653/v1/2021.findings-emnlp.60

Cited by 6 publications

(5 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study also relates to computational work on derivational morphology Cotterell and Schütze, 2018;Deutsch et al, 2018;Hofmann et al, 2020a,b,c) and word segmentation Kann et al, 2016;Ruzsics and Samardžić, 2017;Mager et al, 2019Mager et al, , 2020Seker and Tsarfaty, 2020;Amrhein and Sennrich, 2021). We are the first to systematically evaluate the segmentations of PLM tokenizers on human-annotated gold data.…”

Section: Related Workmentioning

confidence: 92%

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

Hofmann¹,

Schuetze²,

Pierrehumbert³

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

We introduce FLOTA (Few Longest Token Approximation), a simple yet effective method to improve the tokenization of pretrained language models (PLMs). FLOTA uses the vocabulary of a standard tokenizer but tries to preserve the morphological structure of words during tokenization. We evaluate FLOTA on morphological gold segmentations as well as a text classification task, using BERT, GPT-2, and XLNet as example PLMs. FLOTA leads to performance gains, makes inference more efficient, and enhances the robustness of PLMs with respect to whitespace noise.

show abstract

Section: Related Workmentioning

confidence: 92%

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

Hofmann¹,

Schuetze²,

Pierrehumbert³

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

show abstract

“…There is a growing body of work showing how statistical word segmentation methods adversely affect the performance of pretrained language models when dealing with morphologically rich languages like Arabic, Hebrew, Turkish, etc. (Amrhein and Sennrich (2021), Keren et al (2022)). Instead of being completely data-driven, these studies advocate for subword tokenization techniques to be linguistically motivated such that the subwords adhere to morpheme boundaries.…”

Section: Related Workmentioning

confidence: 99%

Impact of Subword Pooling Strategy on Cross-lingual Event Detection

Agarwal¹,

Fincke²,

Jenkins³

et al. 2023

Preprint

View full text Add to dashboard Cite

Pre-trained multilingual language models (e.g., mBERT, XLM-RoBERTa) have significantly advanced the state-ofthe-art for zero-shot cross-lingual information extraction. These language models ubiquitously rely on word segmentation techniques that break a word into smaller constituent subwords. Therefore, all word labeling tasks (e.g. named entity recognition, event detection, etc.), necessitate a pooling strategy that takes the subword representations as input and outputs a representation for the entire word. Taking the task of cross-lingual event detection as a motivating example, we show that the choice of pooling strategy can have a significant impact on the target language performance. For example, the performance varies by up to 16 absolute f1 points depending on the pooling strategy when training in English and testing in Arabic on the ACE task. We carry out our analysis with five different pooling strategies across nine languages in diverse multi-lingual datasets. Across configurations, we find that the canonical strategy of taking just the first subword to represent the entire word is usually sub-optimal. On the other hand, we show that attention pooling is robust to language and dataset variations by being either the best or close to the optimal strategy. For reproducibility, we make our code available at https://github.com/isi-boston/ed-pooling.

show abstract

“…Reducing the number of OOV terms is particularly important for the latter case, as downstream tasks such as classification would incur too high a loss of information if they were just removed. However, it has been argued that languages such as these, as well as agglutinative languages, may be better served with character-level models or small sub-word inventories [ 27 , 28 ], even though sub-word segmentation has reasonable motivation [ 8 ]. Cases like these reinforce the notion that there is no single best solution for language segmentation.…”

Section: Preprocessingmentioning

confidence: 99%

A survey on text classification: Practical perspectives on the Italian language

et al. 2022

View full text Add to dashboard Cite

Text Classification methods have been improving at an unparalleled speed in the last decade thanks to the success brought about by deep learning. Historically, state-of-the-art approaches have been developed for and benchmarked against English datasets, while other languages have had to catch up and deal with inevitable linguistic challenges. This paper offers a survey with practical and linguistic connotations, showcasing the complications and challenges tied to the application of modern Text Classification algorithms to languages other than English. We engage this subject from the perspective of the Italian language, and we discuss in detail issues related to the scarcity of task-specific datasets, as well as the issues posed by the computational expensiveness of modern approaches. We substantiate this by providing an extensively researched list of available datasets in Italian, comparing it with a similarly sought list for French, which we use for comparison. In order to simulate a real-world practical scenario, we apply a number of representative methods to custom-tailored multilabel classification datasets in Italian, French, and English. We conclude by discussing results, future challenges, and research directions from a linguistically inclusive perspective.

show abstract

How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?

Cited by 6 publications

References 39 publications

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

Impact of Subword Pooling Strategy on Cross-lingual Event Detection

A survey on text classification: Practical perspectives on the Italian language

Contact Info

Product

Resources

About