Improving Lemmatization of Non-Standard Languages with Joint Learning

Manjavacas, Enrique; Kádár, Ákos; Kestemont, Mike

doi:10.18653/v1/n19-1153

Cited by 18 publications

(21 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The literature concerning historical text normalisation is considerably larger, and includes approaches based on substitution lists, rules, as well as distance-based, statistical approaches (in particular, character-based neural machine translation) and more recently neural models [4]. To include a modelling of context, normalisation systems can reuse deep-learning architectures originally intended for neural machine translation [4,9] or lemmatisation [6,15].…”

Section: Computational Approachesmentioning

confidence: 99%

“…Results are shown in Table 3. Word normalisation For abbreviation expansion, the neural tagger Pie was trained [15], following a setup already used for Old French [6]. It was trained on an aligned version of the previous D-abb and D-exp (with word separation and line hyphenations normalised) alone, and with the addition of the Oriflamms data.…”

Section: Text Normalisation Approachmentioning

confidence: 99%

See 1 more Smart Citation

Handling Heavily Abbreviated Manuscripts: HTR Engines vs Text Normalisation Approaches

Camps¹,

Vidal-Gorène²,

Vernet³

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Although abbreviations are fairly common in handwritten sources, particularly in medieval and modern Western manuscripts, previous research dealing with computational approaches to their expansion is scarce. Yet abbreviations present particular challenges to computational approaches such as handwritten text recognition and natural language processing tasks. Often, pre-processing ultimately aims to lead from a digitised image of the source to a normalised text, which includes expansion of the abbreviations. We explore different setups to obtain such a normalised text, either directly, by training HTR engines on normalised (i.e., expanded, disabbreviated) text, or by decomposing the process into discrete steps, each making use of specialist models for recognition, word segmentation and normalisation. The case studies considered here are drawn from the medieval Latin tradition.

show abstract

Section: Computational Approachesmentioning

confidence: 99%

Section: Text Normalisation Approachmentioning

confidence: 99%

Handling Heavily Abbreviated Manuscripts: HTR Engines vs Text Normalisation Approaches

Camps¹,

Vidal-Gorène²,

Vernet³

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…For the Vulgate, we follow the traditional segmentation into verses, which amounts to 36,663 documents. Both collections were lemmatized using the neural Latin lemmatizer provided by the software library pie (Manjavacas et al, 2019a).…”

Section: Datasetmentioning

confidence: 99%

Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research

Arevalo¹,

Mellerin²,

Kestemont³

2021

Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities An

Self Cite

View full text Add to dashboard Cite

We report on an inter-annotator agreement experiment involving instances of text reuse focusing on the well-known case of biblical intertextuality in medieval literature. We target the application use case of literary scholars whose aim is to document instances of biblical references in the 'apparatus fontium' of a prospective digital edition. We develop a Bayesian implementation of Cohen's  for multiple annotators that allows us to assess the influence of various contextual effects on the inter-annotator agreement, producing both more robust estimates of the agreement indices as well as insights into the annotation process that leads to the estimated indices. As a result, we are able to produce a novel and nuanced estimation of inter-annotator agreement in the context of intertextuality, exploring the challenges that arise from manually annotating a dataset of biblical references in the writings of Bernard of Clairvaux. Among others, our method is able to unveil the fact that the obtained agreement depends heavily on the biblical source book of the proposed reference, as well as the underlying algorithm used to retrieve the candidate match. Finally, a discussion of the hurdles encountered by annotators supplements the results of the statistical analysis, contributing a qualitative insight into the difficulties involved in the identification of literary text reuse.

show abstract

“…We use Pie [Manjavacas et al 2019] as our main training and tagging software . For training, Pie is used for lemma-and POS-tagging.…”

Section: Set Upmentioning

confidence: 99%

Standardizing linguistic data

Gabay

Clérice

Camps

et al. 2020

Proceedings of the 2nd International Conference on Digital Tools &Amp; Uses Congress

View full text Add to dashboard Cite

Improving Lemmatization of Non-Standard Languages with Joint Learning

Cited by 18 publications

References 26 publications

Handling Heavily Abbreviated Manuscripts: HTR Engines vs Text Normalisation Approaches

Handling Heavily Abbreviated Manuscripts: HTR Engines vs Text Normalisation Approaches

Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research

Standardizing linguistic data

Contact Info

Product

Resources

About