The notion of a “lemma”

Knowles, Gerry; Don, Zuraidah Mohd

doi:10.1075/ijcl.9.1.04kno

Cited by 25 publications

(10 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is therefore also common to annotate words with other sorts of labels, instead of attempting an explicit respelling. Lemmas are often used in this respect Van der Voort van der Kleij, 2005: a lemma is a normalized label, which unambiguously links words to the same entry (headword) in a lexical resource, such as a dictionary, if they only differ in inflection or spelling (Knowles and Mohd Don, 2004).…”

Section: Related Researchmentioning

confidence: 99%

“…7 Each token in the corpora has been annotated with a normalized dictionary headform or lemma in a present-day spelling. Broadly speaking, lemmatization allows us to abstract over individual instances of tokens which only differ in inflection or orthography (Knowles and Mohd Don, 2004). For historical corpora, this has interesting applications in the context of database searching, (diachronic) topic modelling or stylometry.…”

Section: Data Setsmentioning

confidence: 99%

See 1 more Smart Citation

Lemmatization for variation-rich languages using deep learning

Kestemont

Pauw

Nie

et al. 2016

Digital Scholarship Humanities

View full text Add to dashboard Cite

In this article, we describe a novel approach to sequence tagging for languages that are rich in (e.g. orthographic) surface variation. We focus on lemmatization, a basic step in many processing pipelines in the Digital Humanities. While this task has long been considered solved for modern languages such as English, there exist many (e.g. historic) languages for which the problem is harder to solve, due to a lack of resources and unstable orthography. Our approach is based on recent advances in the field of 'deep' representation learning, where neural networks have led to a dramatic increase in performance across several domains. The proposed system combines two approaches: on the one hand, we apply temporal convolutions to model the orthography of input words at the character level; secondly, we use distributional word embeddings to represent the lexical context surrounding the input words. We demonstrate how this system reaches state-ofthe-art performance on a number of representative Middle Dutch data sets, even without corpus-specific parameter tuning.

show abstract

Section: Related Researchmentioning

confidence: 99%

Section: Data Setsmentioning

confidence: 99%

Lemmatization for variation-rich languages using deep learning

Kestemont

Pauw

Nie

et al. 2016

Digital Scholarship Humanities

View full text Add to dashboard Cite

show abstract

“…Lemmatization is the task of mapping a token to its corresponding dictionary head-form to allow downstream applications to abstract away from orthographic and inflectional variation (Knowles and Mohd Don, 2004). While lemmatization is considered to be solved for analytic and resourcerich languages such as English, it remains an open challenge for morphologically complex (e.g.…”

Section: Introductionmentioning

confidence: 99%

Improving Lemmatization of Non-Standard Languages with Joint Learning

Manjavacas¹,

Kádár²,

Kestemont³

2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Lemmatization of standard languages is concerned with (i) abstracting over morphological differences and (ii) resolving token-lemma ambiguities of inflected words in order to map them to a dictionary headword. In the present paper we aim to improve lemmatization performance on a set of non-standard historical languages in which the difficulty is increased by an additional aspect (iii): spelling variation due to lacking orthographic standards. We approach lemmatization as a stringtransduction task with an encoder-decoder architecture which we enrich with sentence context information using a hierarchical sentence encoder. We show significant improvements over the state-of-the-art when training the sentence encoder jointly for lemmatization and language modeling. Crucially, our architecture does not require POS or morphological annotations, which are not always available for historical corpora. Additionally, we also test the proposed model on a set of typologically diverse standard languages showing results on par or better than a model without enhanced sentence representations and previous state-of-the-art systems. Finally, to encourage future work on processing of non-standard varieties, we release the dataset of non-standard languages underlying the present study, based on openly accessible sources.

show abstract

“…Knowles and Mohd Don 2004). According to lexicographical recommendations, all forms which naturally come to mind to users when searching a dictionary should function as headwords.…”

Section: Citation-forms In Swahili Dictionariesmentioning

confidence: 99%

Revisiting Lemma Lists in Swahili Dictionaries

Wójtowicz

2017

LEXI

View full text Add to dashboard Cite

When compiling a dictionary, a lexicographer has a set of decisions to make -starting with drawing up a lemma list to such issues as formatting a dictionary entry. Relying on corpus data while designing a lemma list and describing entries is standard in present lexicography, but there are still decisions -like the choice of a lemma or how to treat derivatives -that are often intuition-based. This article aims to investigate whether decisions put forward in Swahili dictionaries comply with users' expectations. We analyse log files from the new Swahili-Polish dictionary to investigate why looking up words goes wrong, and evaluate the choice of a lemma and the treatment of derivatives in Swahili dictionaries. Based on such data we intend to expand or modify the existing electronic dictionary to adapt to users' level of grammar and dictionary structure knowledge. During this research we identified a list of lemma lacuna that cause the majority of unsuccessful Swahili searches. The study shows that users know and understand the lemmatisation strategy of the dictionary but also reveals which word forms cause the most problems and how the lemma list of Swahili dictionaries could be expanded. Keywords: DICTIONARY USER RESEARCH, LOG FILES ANALYSIS, SWAHILI-POLISH DICTIONARY, LEMMA LIST, DERIVATIVESOpsomming: Die herbesoek van lemmalyste in Swahili-woordeboeke. Wanneer 'n woordeboek saamgestel word, moet 'n leksikograaf 'n reeks besluite neem -van die opstel van 'n lemmalys, tot kwessies soos die formatering van 'n woordeboekinskrywing. Om staat te maak op korpusdata wanneer 'n lemmalys opgestel en inskrywings beskryf word, is standaardpraktyk in die huidige leksikografie, maar daar is steeds dikwels besluite -soos die keuse van 'n lemma of hoe om afleidings te hanteer -wat op intuïsie gebaseer is. Hierdie artikel beoog om te ondersoek of besluite wat in Swahili-woordeboeke geneem is, voldoen aan gebruikers se verwagtings. Ons analiseer loglêers van die nuwe Swahili-Poolse woordeboek om te ondersoek waarom die opsoek van woorde skeefloop, en evalueer die lemmakeuse en die hantering van afleidings in Swahili-woordeboeke. Ons beoog om die bestaande elektroniese woordeboek op grond van hierdie data uit te brei of te wysig om aan te pas by gebruikers se vlak van kennis ten opsigte van grammatika en woordeboekstruktuur. Tydens hierdie navorsing het ons 'n lys van leemtes ten opsigte van lemmas geïdentifiseer wat die meerderheid van onsuksesvolle Swahili-soektogte veroorsaak. Die navorsing toon dat gebruikers die woordeboek se lemmatiseringstrategie ken en verstaan, maar openbaar ook watter woordvorme die meeste probleme veroorsaak en hoe die lemmalys van Swahiliwoordeboeke uitgebrei kan word.

show abstract

The notion of a “lemma”

Cited by 25 publications

References 2 publications

Lemmatization for variation-rich languages using deep learning

Lemmatization for variation-rich languages using deep learning

Improving Lemmatization of Non-Standard Languages with Joint Learning

Revisiting Lemma Lists in Swahili Dictionaries

Contact Info

Product

Resources

About