Retrieval in text collections with historic spelling using linguistic and spelling variants

Ernst-Gerlach, Andrea; Fuhr, Norbert

doi:10.1145/1255175.1255242

Cited by 31 publications

(23 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most approaches to historical spelling variation for correction and/or text retrieval attempt to model the historical typographical variation observed by means of rules, either created manually or derived (semi-)automatically ( [7] for English, [8], [9], [10] for German and [11] for Dutch). These authors typically work on older, pre-standardization era language variants than we do here.…”

Section: Historical Text Collections: Prior Workmentioning

confidence: 99%

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Reynaert

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

Abstract. This paper proposes a non-interactive system for reducing the level of OCR-induced typographical variation in large text collections, contemporary and historical. Text-Induced Corpus Clean-up or ticcl (pronounce 'tickle') focuses on high-frequency words derived from the corpus to be cleaned and gathers all typographical variants for any particular focus word that lie within the predefined Levenshtein distance (henceforth: ld). Simple text-induced filtering techniques help to retain as many as possible of the true positives and to discard as many as possible of the false positives. ticcl has been evaluated on a contemporary OCR-ed Dutch text corpus and on a corpus of historical newspaper articles, whose OCR-quality is far lower and which is in an older Dutch spelling. Representative samples of typographical variants from both corpora have allowed us not only to properly evaluate our system, but also to draw effective conclusions towards the adaptation of the adopted correction mechanism to OCR-error resolution. The performance scores obtained up to ld 2 mean that the bulk of undesirable OCR-induced typographical variation present can fully automatically be removed.

show abstract

Section: Historical Text Collections: Prior Workmentioning

confidence: 99%

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Reynaert

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

show abstract

“…Previous work [35,36,69] addressed the spelling variation problem using techniques from cross language information retrieval (CLIR). In [69], Koolen et al proposed a crosslanguage approach to historic document retrieval.…”

Section: Searching With the Awareness Of Terminology Changesmentioning

confidence: 99%

“…A rule-based method for modernizing historic languages, and the retrieval of historic documents using cross-language information retrieval techniques are proposed. In [35,36], Ernst-Gerlach and Fuhr used probabilistic rule-based approaches to handling term variants when searching historic texts. In this case, a user can search using queries in contemporary language and the issued queries are translated into an old spelling possibly unknown to the user, which is similar to query expansion.…”

Section: Searching With the Awareness Of Terminology Changesmentioning

confidence: 99%

Time-aware approaches to information retrieval

Kanhabua

2012

SIGIR Forum

View full text Add to dashboard Cite

In this thesis, we address major challenges in searching temporal document collections. In such collections, documents are created and/or edited over time. Examples of temporal document collections are web archives, news archives, blogs, personal emails and enterprise documents. Unfortunately, traditional IR approaches based on term-matching only can give unsatisfactory results when searching temporal document collections. The reason for this is twofold: the contents of documents are strongly time-dependent, i.e., documents are about events happened at particular time periods, and a query representing an information need can be time-dependent as well, i.e., a temporal query.Our contributions in this thesis are different time-aware approaches within three topics in IR: content analysis, query analysis, and retrieval and ranking models. In particular, we aim at improving the retrieval effectiveness by 1) analyzing the contents of temporal document collections, 2) performing an analysis of temporal queries, and 3) explicitly modeling the time dimension into retrieval and ranking.Leveraging the time dimension in ranking can improve the retrieval effectiveness if information about the creation or publication time of documents is available. In this thesis, we analyze the contents of documents in order to determine the time of non-timestamped documents using temporal language models. We subsequently employ the temporal language models for determining the time of implicit temporal queries, and the determined time is used for re-ranking search results in order to improve the retrieval effectiveness.We study the effect of terminology changes over time and propose an approach to handling terminology changes using time-based synonyms. In addition, we propose different methods for predicting the effectiveness of temporal queries, so that a particular query enhancement technique can be performed to improve the overall performance. When the time dimension is incorporated into ranking, documents will be ranked according to both textual and temporal similarity. In this case, time uncertainty should also be taken into account. Thus, we propose a ranking model that considers the time uncertainty, and improve ranking by combining multiple features using learning-to-rank techniques.Through extensive evaluation, we show that our proposed time-aware approaches outperform traditional retrieval methods and improve the retrieval effectiveness in searching temporal document collections.

show abstract

“…A special case of evolution, outdated spellings of the same term, has been addressed in [9] where a rule based method is used for deriving spelling variations that are later used for information retrieval. In order to overcome a larger class of issues caused by language evolution in historic collections, it is necessary to develop methods and models designed especially for this purpose.…”

Section: Introductionmentioning

confidence: 99%

“…In order to overcome a larger class of issues caused by language evolution in historic collections, it is necessary to develop methods and models designed especially for this purpose. Due to the size of the collections, an explicit modeling of semantics, such as those found in [9] is not possible. Therefore we use word sense discrimination as a statistical method to learn the models directly from historic archives [22].…”

Section: Introductionmentioning

confidence: 99%

Using word sense discrimination on historic document collections

Tahmasebi

Niklas

Theuerkauf

et al. 2010

Proceedings of the 10th Annual Joint Conference on Digital Libraries

View full text Add to dashboard Cite

Word sense discrimination is the first, important step towards automatic detection of language evolution within large, historic document collections. By comparing the found word senses over time, we can reveal and use important information that will improve understanding and accessibility of a digital archive. Algorithms for word sense discrimination have been developed while keeping today's language in mind and have thus been evaluated on well selected, modern datasets. The quality of the word senses found in the discrimination step has a large impact on the detection of language evolution. Therefore, as a first step, we verify that word sense discrimination can successfully be applied to digitized historic documents and that the results correctly correspond to word senses. Because accessibility of digitized historic collections is influenced also by the quality of the optical character recognition (OCR), as a second step we investigate the effects of OCR errors on word sense discrimination results. All evaluations in this paper are performed on The Times Archive, a collection of newspaper articles from 1785 − 1985.

show abstract

Retrieval in text collections with historic spelling using linguistic and spelling variants

Cited by 31 publications

References 7 publications

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Time-aware approaches to information retrieval

Using word sense discrimination on historic document collections

Contact Info

Product

Resources

About