Automatic detection and correction of spelling errors in a large data base

Abstract

On‐line bibliographic search systems tend to increase the visibility of spelling errors through the use of indexes of unique terms; even low error rates in a data base can result in large numbers of misspelled terms in these indexes. This article describes the techniques used to detect and correct spelling errors in the data base of Chemical Abstracts Service. A computer program for spelling error detection achieves a high level of performance using hashing techniques for dictionary look‐up and compression. Heuristic procedures extend the dictionary and increase the proportion of misspelled words in the words flagged. Automatic correction procedures are applied only to words which are known to be misspelled; other corrections are performed manually during the normal editorial cycle. The constraints imposed on the selection of a spelling error detection technique by a complex data base, human factors, and high‐volume production are discussed.

Probabilistic retrieval of OCR degraded text using N-grams

“…This approach uses n-grams to discover words that match and "nearly" match target terms, then add these additional terms to the original query. This approach has wide appeal since it could be largely language independent and could be applied to various concept (word) representations such as phonemes, soundex codes [14,13] or for spelling correction [12], using differing retrieval engines [2], or as a means to summarize the content of a document [3].…”

Highlights: Language- and domain-independent automatic indexing terms for abstracting

“…A variety of spelling-related applications have been proposed and examined: n-grams have been used to correct Morse code (McElwain & Evens, 1962); flag possible typing errors (Morris&Cherry, 1975;Zamora, 1980;Zamora et al, 1981); perform general spelling correction (Angell, 1983;Ullmann, 1977); aid in optical character recognition (OCR) classification (Hussain & Donaldson, 1974;Shinghal, Rosenberg, & Toussaint, 1978); correct spelling errors in OCR after classification (Cornew, 1968;Damerau, 1964;Thomas & Kassler, 1967); and to do both detection and correction of OCR spelling errors (Hanson et al, 1976;Hull & Srihari, 1982;Neuhoff, 1975;Riseman & Hanson, 1974;Vossler & Branston, 1964). An overview of spelling-related applications and results may be found in the excellent reviews by Suen ( 1979) and Kukich ( 1992).…”

Calidad de la indización e incidencia de errores en la base de datos ECOSOC

“…El análisis se ha centrado en la detecci6n de campos vacíos, datos mal situados, referencias duplicadas, normalizaci6n de las reglas de escritura (puntos, abreviaturas, etc.) y en la detecci6n de errores de ortografía y tipografía (13)(14)(15). No se ha hecho distinci6n entre estos dos últimos tipos de errores (16).…”