A Memory-Based Lemmatizer for Ancient Greek

Abstract

In this paper we present the lemmatizer that we developed for Ancient Greek: GLEM. As far as we know, GLEM is the rst publicly available lemmatizer for Ancient Greek that uses POS information to disambiguate and that also assigns output to unseen words, words that are not yet in the lexicon. As the basis for the lemmatizer we used an existing memorybased learning tool, Frog, that was originally developed for Dutch and that we converted to work for Ancient Greek. As the results of Frog on Ancient Greek were rather modest, we used Frog to create a smarter lemmatizer, GLEM, that uses a lexicon look up in addition to the memory-based tool Frog. We evaluate and compare the performance of GLEM against the Frog lemmatizer and the already existing CLTK lemmatizer and observe that GLEM achieves the highest accuracy of 93% on an unseen test corpus sample. GLEM's look up component overcomes the di culty of a relative small training set in combination with a morphologically rich language, while the memory-based learning component enables GLEM to handle unknown words.

Lemmatization for Ancient Greek

“…These scores suggest that Diorisis outperforms other corpora and lemmatizers on both samples. glem scores surprisingly low compared to the reported accuracy of 93% on an unseen text reported by its developers (Bary et al 2017).…”

“…glem chooses the lemma of the most frequent word form/part-of-speech combination according to the proiel lexicon whenever a disambiguation purely based on part of speech is not possible. Bary et al (2017) show that glem outperforms both a lemmatizer based on Frog and the cltk lemmatizer on an unseen text, with an accuracy of 93.0% vs. 75.6% and 76.6%, respectively, making it the state-of-the-art system for Ancient Greek in 2017.…”

“…Several lemmatizers for Ancient Greek are currently available to the scholarly community; see Bary et al (2017) for an overview. Because Bary et al show that their system, glem (short, presumably, for Greek Lemmatizer), outperforms other state-of-the-art lemmatizers for Ancient Greek, we decided to focus on comparing glem with newer lemmatizers which have been developed since glem was created, and which are described in this section.…”

“…The glem lemmatizer (Bary et al 2017) is built from Frog (Hendrickx et al 2016), an open-source natural language processing tool based on memorybased algorithms (Daelemans & van den Bosch 2010). This machine-learning component is combined with a lexicon look-up in order to use part-of-speech information to disambiguate between multiple lemmas.…”

Annotating Speech, Attitude and Perception Reports

“…As it was in betacode we converted it into unicode (utf8) using a converter created by the Classical Language Toolkit (Johnson and others, 2016). 7 As mentioned before, we combine the manual reports annotation with automated POS-tagging and lemmatization (Bary et al, 2017), which we developed independently and which is open source.…”