A Suffix Based Part-of-Speech Tagger for Turkish

Abstract

In this paper, we present a stochastic part-of-speech tagger for Turkish. The tagger is primarily developed for information retrieval purposes, but it can as well serve as a light-weight PoS tagger for other purposes. The tagger uses a well-established Hidden Markov model of the language with a closed lexicon that consists of fixed number of letters from the word endings. We have considered seven different lengths of word endings against 30 training corpus sizes. Bestcase accuracy obtained is 90.2% with 5 characters. The main contribution of this paper is to present a way of constructing a closed vocabulary for part-of-speech tagging effort that can be useful for highly inflected languages like Turkish, Finnish, Hungarian, Estonian, and Czech.

Turkish PoS Tagging by Reducing Sparsity with Morpheme Tags in Small Datasets

“…We believe that morphological features of the context words will be also informative in morpheme tagging task because Eryigit et al [9] shows that using Table 8. Comparison of our model with suffix based tagger [19] and the perceptron algorithm [16] on the datasets obtained from Metu Sabancı Turkish Treebank.…”

“…Train Set Size Test Set Size Accuracy Suffix-based tagger [19] 5025 1017 84.25% Perceptron [16] 5025 1017 85.15% HMM with the last morpheme tag 5025 1017 88.98% Suffix-based tagger [19] 18205 1017 88.90% Perceptron [16] 18205 1017 86.71% HMM with the last morpheme tag 18205 1017 90.95%…”

“…Dincer et al [19] formulate HMMs by emitting suffixes rather than emitting words in order to mitigate the sparsity. However, they do not use any morpheme tags.…”

“…It shows that even though terminal punctuation plays an important role in PoS tagging, behaving each sentence as a single HMM by assuming that sentences are independent from each other leads to a slight increase in the accuracy. We compared our model with Sak et al [16] and Dincer et al [19] on Metu Sabancı Turkish Treebank. We used the last 5 letters of each word with a second order HMM to implement the suffix based tagger by Dincer et al [19], since their model gives the best scores for the last 5 letters.…”

A hybrid Statistical Approach to Stemming in Turkish: An Agglutinative Language

“…Before starting the analysis, the word is PoS tagged within the context of a sentence using suffix-based hidden Markov model [27]. In this example, we assume that PoS tag of the word is noun.…”