Generating Search Term Variants for Text Collections with Historic Spellings

Ernst-Gerlach, Andrea; Fuhr, Norbert

doi:10.1007/11735106_6

Cited by 24 publications

(11 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The need for handling morphological variation has been acknowledged in previous studies on historical retrieval. For example, Ernst‐Gerlach and Fuhr (, ) showed that a high coverage of German word form variants in a historical collection was achieved by first generating all modern inflectional variants of the query words, and then generating the historical variants of the inflectional forms. Even a combination of a modern stemmer and the historical variant generation performed well (Ernst‐Gerlach & Fuhr, ).…”

Section: Discussionmentioning

confidence: 99%

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Järvelin

Keskustalo

Sormunen

et al. 2015

Asso for Info Science & Tech

View full text Add to dashboard Cite

The aim of the study was to test whether query expansion by approximate string matching methods is beneficial in retrieval from historical newspaper collections in a language rich with compounds and inflectional forms (Finnish). First, approximate string matching methods were used to generate lists of index words most similar to contemporary query terms in a digitized newspaper collection from the 1800s. Top index word variants were categorized to estimate the appropriate query expansion ranges in the retrieval test. Second, the effectiveness of approximate string matching methods, automatically generated inflectional forms, and their combinations were measured in a Cranfield-style test. Finally, a detailed topic-level analysis of test results was conducted. In the index of historical newspaper collection the occurrences of a word typically spread to many linguistic and historical variants along with optical character recognition (OCR) errors. All query expansion methods improved the baseline results. Extensive expansion of around 30 variants for each query word was required to achieve the highest performance improvement. Query expansion based on approximate string matching was superior to using the inflectional forms of the query words, showing that coverage of the different types of variation is more important than precision in handling one type of variation. IntroductionDigitization is a good way to preserve cultural heritage documents and make them widely accessible for researchers and the general public. Cultural institutions are aware of this potential and often consider digitization of their cultural heritage collections as an obligation. Consequently, the quantity of digitized historical documents available is constantly growing. Transforming print cultural heritage collections into digital resources accessible and searchable through modern information and communication technologies requires that the digitized document images are transformed into digital text through optical character recognition (OCR). While OCR can currently reach over 99% accuracy in recognition of characters from high-quality images of original documents with a simple book layout, the accuracy for historical newspapers is lower than that. OCR quality is dependent on the environment and the condition of the original documents: print and paper quality, typefaces, and layout complexity affect the accuracy of the result. Generally, the older the newspaper is, the lower the accuracy rate is likely to be. Holley (2009) reported raw character recognition accuracy rates varying from 71% to 98% in a sample of digitized newspapers from 1803-1954, the lowest rate indicating almost every third character being erroneously recognized and virtually all words containing errors. Even a 98% accuracy rate results in an error in, on average, every sixth word in Finnish text (with an average word length of around eight characters), if the errors are evenly distributed. Such error rates may lead to a quadrupling of the number of unique index words and sign...

show abstract

Section: Discussionmentioning

confidence: 99%

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Järvelin

Keskustalo

Sormunen

et al. 2015

Asso for Info Science & Tech

View full text Add to dashboard Cite

show abstract

“…Previous work [35,36,69] addressed the spelling variation problem using techniques from cross language information retrieval (CLIR). In [69], Koolen et al proposed a crosslanguage approach to historic document retrieval.…”

Section: Searching With the Awareness Of Terminology Changesmentioning

confidence: 99%

“…A rule-based method for modernizing historic languages, and the retrieval of historic documents using cross-language information retrieval techniques are proposed. In [35,36], Ernst-Gerlach and Fuhr used probabilistic rule-based approaches to handling term variants when searching historic texts. In this case, a user can search using queries in contemporary language and the issued queries are translated into an old spelling possibly unknown to the user, which is similar to query expansion.…”

Section: Searching With the Awareness Of Terminology Changesmentioning

confidence: 99%

“…In this case, a user can search using queries in contemporary language and the issued queries are translated into an old spelling possibly unknown to the user, which is similar to query expansion. As explained in [35], there are two ways to perform query expansion: an expansion of query and an expansion of index. In the first case, a set of rules is automatically constructed for mapping historic terms into modern terms.…”

Section: Searching With the Awareness Of Terminology Changesmentioning

confidence: 99%

See 1 more Smart Citation

Time-aware approaches to information retrieval

Kanhabua

2012

SIGIR Forum

View full text Add to dashboard Cite

In this thesis, we address major challenges in searching temporal document collections. In such collections, documents are created and/or edited over time. Examples of temporal document collections are web archives, news archives, blogs, personal emails and enterprise documents. Unfortunately, traditional IR approaches based on term-matching only can give unsatisfactory results when searching temporal document collections. The reason for this is twofold: the contents of documents are strongly time-dependent, i.e., documents are about events happened at particular time periods, and a query representing an information need can be time-dependent as well, i.e., a temporal query.Our contributions in this thesis are different time-aware approaches within three topics in IR: content analysis, query analysis, and retrieval and ranking models. In particular, we aim at improving the retrieval effectiveness by 1) analyzing the contents of temporal document collections, 2) performing an analysis of temporal queries, and 3) explicitly modeling the time dimension into retrieval and ranking.Leveraging the time dimension in ranking can improve the retrieval effectiveness if information about the creation or publication time of documents is available. In this thesis, we analyze the contents of documents in order to determine the time of non-timestamped documents using temporal language models. We subsequently employ the temporal language models for determining the time of implicit temporal queries, and the determined time is used for re-ranking search results in order to improve the retrieval effectiveness.We study the effect of terminology changes over time and propose an approach to handling terminology changes using time-based synonyms. In addition, we propose different methods for predicting the effectiveness of temporal queries, so that a particular query enhancement technique can be performed to improve the overall performance. When the time dimension is incorporated into ranking, documents will be ranked according to both textual and temporal similarity. In this case, time uncertainty should also be taken into account. Thus, we propose a ranking model that considers the time uncertainty, and improve ranking by combining multiple features using learning-to-rank techniques.Through extensive evaluation, we show that our proposed time-aware approaches outperform traditional retrieval methods and improve the retrieval effectiveness in searching temporal document collections.

show abstract

“…However, the same experiments showed lower performance on unknown wordforms [3], specially in some periods (time in history) the rules were varying. The rulebased approach was widely used in Information Retrieval applied for normalizing historical language data [6,1,19,12].…”

Section: Introductionmentioning

confidence: 99%

Normalizing historical orthography for OCR historical documents using LSTM

Azawi

Afzal

Breuel

2013

Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing

View full text Add to dashboard Cite

Historical text presents numerous challenges for contemporary different techniques, e.g. information retrieval, OCR and POS tagging. In particular, the absence of consistent orthographic conventions in historical text presents difficulties for any system which requires reference to a fixed lexicon accessed by orthographic form. For example, language modeling or retrieval engine for historical text which is produced by OCR systems, where the spelling of words often differ in various way, e.g. one word might have different spellings evolved over time. It is very important to aid those techniques with the rules for automatic mapping of historical wordforms. In this paper, we propose a new technique to model the target modern language by means of a recurrent neural network with long-short term memory architecture. Because the network is recurrent, the considered context is not limited to a fixed size especially due to memory cells which are designed to deal with long-term dependencies. In the set of experiments conducted on the Luther bible database and transform wordforms from Early New High German (ENHG) 14 th -16 th centuries to the corresponding modern wordforms in New High German (NHG). We compare our proposed supervised model LSTM to various methods for computing word alignments using statistical, heuristic models. Our new proposed LSTM outperforms the other three state-of-the-art methods. The evaluation shows the accuracy of our model on the known wordforms is 93.90% and on the unknown wordforms is 87.95%, while the accuracy of the existing state-of-the-art combined approach of the wordlist-based and rule-based normalization models is 92.93% for known and 76.88% for unknown tokens. Our proposed LSTM model outperforms on normalizing the modern wordform to historical wordform. The performance on seen tokens is 93.4%, while for unknown tokens is 89.17%.

show abstract

Generating Search Term Variants for Text Collections with Historic Spellings

Cited by 24 publications

References 5 publications

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Time-aware approaches to information retrieval

Normalizing historical orthography for OCR historical documents using LSTM

Contact Info

Product

Resources

About