The Identification of Spelling Variants in English and German Historical Texts: Manual or Automatic?

Pilz, Thomas; Ernst-Gerlach, Andrea; Kempken, Sebastian; Rayson, Paul; Archer, Dawn

doi:10.1093/llc/fqm044

Cited by 17 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is the case of research using NER for materials with disparate qualities of digitization and OCR, non-European or classical languages, or collections featuring spelling variations (Alex et al, 2012;Batjargal et al, 2014;Neudecker et al, 2014;Nagai et al, 2015;Erdmann et al, 2016;Kettunen et al, 2016). Previous research within Digital Humanities has also tackled the related problem of text geoparsing, leveraging NER methods for recognizing place references in text, often together with other heuristics for the complete resolution of place references into gazetteer entries and/or geographical coordinates (Rayson et al, 2006;Baron and Rayson, 2008;Pilz et al, 2008;Grover et al, 2010;Freire et al, 2011;Gregory and Hardie, 2011;Brown et al, 2012;Alex et al, 2015;Gregory et al, 2015;Santos et al, 2015a,b;Wing, 2015;Clifford et al, 2016).…”

Section: Ner In Historical Documentsmentioning

confidence: 99%

Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora

Won

Murrieta-Flores

Martins

2018

Front. Digit. Humanit.

View full text Add to dashboard Cite

The field of Spatial Humanities has advanced substantially in the past years. The identification and extraction of toponyms and spatial information mentioned in historical text collections has allowed its use in innovative ways, making possible the application of spatial analysis and the mapping of these places with geographic information systems. For instance, automated place name identification is possible with Named Entity Recognition (NER) systems. Statistical NER methods based on supervised learning, in particular, are highly successful with modern datasets. However, there are still major challenges to address when dealing with historical corpora. These challenges include language changes over time, spelling variations, transliterations, OCR errors, and sources written in multiple languages among others. In this article, considering a task of place name recognition over two collections of historical correspondence, we report an evaluation of five NER systems and an approach that combines these through a voting system. We found that although individual performance of each NER system was corpus dependent, the ensemble combination was able to achieve consistent measures of precision and recall, outperforming the individual NER systems. In addition, the results showed that these NER systems are not strongly dependent on preprocessing and translation to Modern English.

show abstract

Section: Ner In Historical Documentsmentioning

confidence: 99%

Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora

Won

Murrieta-Flores

Martins

2018

Front. Digit. Humanit.

View full text Add to dashboard Cite

show abstract

“…Koolen et al considered the spelling and pronunciation differences between ancient and modern Dutch [20], while Gotscharek et al [21] and Hauser et al [22] considered the spelling differences and variations between modern and archaic German. Pilz et al considered spelling variations of English and German historical texts [23]. In general, the main challenge for historical European languages like Dutch, English and German is the spelling variants.…”

Section: Adopting Cross-language and Cross-chronological Information mentioning

confidence: 99%

Cross-Lingual and Cross-Chronological Information Access to Multilingual Historical Documents

Batjargal¹

2018

Multilingualism and Bilingualism

View full text Add to dashboard Cite

In this chapter, we present our work in realizing information access across different languages and periods. Nowadays, digital collections of historical documents have to handle materials written in many different languages in different time periods. Even in a particular language, there are significant differences over time in terms of grammar, vocabulary and script. Our goal is to develop a method to access digital collections in a wide range of periods from ancient to modern. We introduce an information extraction method for digitized ancient Mongolian historical manuscripts for reducing labour-intensive analysis. The proposed method performs computerized analysis on Mongolian historical documents. Named entities such as personal names and place names are extracted by employing support vector machine. The extracted named entities are utilized to create a digital edition that reflects an ancient Mongolian historical manuscript written in traditional Mongolian script. The Text Encoding Initiative guidelines are adopted to encode the named entities, transcriptions and interpretations of ancient words. A web-based prototype system is developed for utilizing digital editions of ancient Mongolian historical manuscripts as scholarly tools. The proposed prototype has the capability to display and search traditional Mongolian text and its transliteration in Latin letters along with the highlighted named entities and the scanned images of the source manuscript.

show abstract

“…This approach outperformed different combinations of stemming, edit distance, and word form generation, showing that handling both inflection and historical variation is important for highly inflectional languages. Pilz, Ernst‐Gerlach, Kempken, Rayson, and Archer () found that automatic approaches to historical variant generation can reproduce manually generated gold standard rules quite well and may also capture variation that is not discovered manually. They argued for generic letter‐replacement heuristics for Germanic languages and showed that an edit distance variant where the edit costs were learned from German historical corpus outperformed the standard edit distance of English historical data.…”

Section: Related Researchmentioning

confidence: 99%

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Järvelin

Keskustalo

Sormunen

et al. 2015

Asso for Info Science & Tech

View full text Add to dashboard Cite

The aim of the study was to test whether query expansion by approximate string matching methods is beneficial in retrieval from historical newspaper collections in a language rich with compounds and inflectional forms (Finnish). First, approximate string matching methods were used to generate lists of index words most similar to contemporary query terms in a digitized newspaper collection from the 1800s. Top index word variants were categorized to estimate the appropriate query expansion ranges in the retrieval test. Second, the effectiveness of approximate string matching methods, automatically generated inflectional forms, and their combinations were measured in a Cranfield-style test. Finally, a detailed topic-level analysis of test results was conducted. In the index of historical newspaper collection the occurrences of a word typically spread to many linguistic and historical variants along with optical character recognition (OCR) errors. All query expansion methods improved the baseline results. Extensive expansion of around 30 variants for each query word was required to achieve the highest performance improvement. Query expansion based on approximate string matching was superior to using the inflectional forms of the query words, showing that coverage of the different types of variation is more important than precision in handling one type of variation. IntroductionDigitization is a good way to preserve cultural heritage documents and make them widely accessible for researchers and the general public. Cultural institutions are aware of this potential and often consider digitization of their cultural heritage collections as an obligation. Consequently, the quantity of digitized historical documents available is constantly growing. Transforming print cultural heritage collections into digital resources accessible and searchable through modern information and communication technologies requires that the digitized document images are transformed into digital text through optical character recognition (OCR). While OCR can currently reach over 99% accuracy in recognition of characters from high-quality images of original documents with a simple book layout, the accuracy for historical newspapers is lower than that. OCR quality is dependent on the environment and the condition of the original documents: print and paper quality, typefaces, and layout complexity affect the accuracy of the result. Generally, the older the newspaper is, the lower the accuracy rate is likely to be. Holley (2009) reported raw character recognition accuracy rates varying from 71% to 98% in a sample of digitized newspapers from 1803-1954, the lowest rate indicating almost every third character being erroneously recognized and virtually all words containing errors. Even a 98% accuracy rate results in an error in, on average, every sixth word in Finnish text (with an average word length of around eight characters), if the errors are evenly distributed. Such error rates may lead to a quadrupling of the number of unique index words and sign...

show abstract

The Identification of Spelling Variants in English and German Historical Texts: Manual or Automatic?

Cited by 17 publications

References 7 publications

Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora

Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora

Cross-Lingual and Cross-Chronological Information Access to Multilingual Historical Documents

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Contact Info

Product

Resources

About