A hybrid Statistical Approach to Stemming in Turkish: An Agglutinative Language

Abstract

Finding Stem is a complicated and important issue for agglutinative languages like Turkish where theoretically infinite number of surface forms can be obtained from a single lexeme. Both analytical and statistical approaches have been tried for stemming Turkish words. Two main problems that become apparent with these approaches are the involvement of a dictionary which enforces the assumption of closed vocabulary and the disambiguation of the actual stem among the numerous candidates. Here, we present a method that exploits the simple fact that nouns and verbs have different suffix patterns. We also use statistical methods which are used for stripping off the suffixes. Based on the suffix pattern PoS is determined, which then enables the decision for the stem boundary. Thus, the presented stemming technique that does not employ a regular dictionary, is a remedy for the disambiguation problem. The performance rate of the method on golden standard PoS tagged METU-Sabancı Turkish Treebank is found to be 93.83%.

Advancing natural language processing (NLP) applications of morphologically rich languages with bidirectional encoder representations from transformers (BERT): an empirical case study for Turkish

“…According to the study, it is possible to derive about 1.5 million different words from a Noun [masa (table)] and from a Verb [oku (read)] only with the use of derivational morphemes [40]. The morphological structure of Turkish word is shown in Figure 1 [41].…”

“…Some samples for morphological productivity of Turkish language are provided in Table 1 [41]. As it is obvious from Table 1, the number of suffixes and their imaginable combinations that can be added to a word generate a serious language analysis problem to obtain actual stem from possible derivations.…”

“…However, the two methods are not interchangeable and it should be carefully examined which one is better for the corresponding language problem. For example, the Turkish words göz (eye), gözlük (eyeglasses), gözlükçü (optician) and gözlem (observation) may all be stemmed from a single word "göz (eye)" losing the semantical information [41,46]. Interestingly, it is apparent that the given Turkish words have distinct English equivalents and this may be a concise comparison of two languages in terms of analysis complexity.…”