Collection and characterization of spelling errors in scientific and scholarly text

Abstract

The SPEEDCOP (SPElling Error Detection Correction Project) project recently completed at Chemical Abstracts Service (CAS) extracted over 50,000 misspellings from approximately 25,000,000 words of text from seven scientific and scholarly databases. The misspellings were automatically classified and the error types analyzed. The results, which were consistent over the different databases, showed that the expected incidence of misspelling is 0.2%, that 90-95% of spelling errors have only a single mistake, that substitution is homogeneous while transposition is heterogeneous, that omission is the commonest type of misspelling, and that inadvertent doubling of a letter is the most important cause of insertion errors. The more frequently a letter occurs in the text, the more likely it is to be involved in a spelling error. Most misspellings collected by SPEEDCOP are of the type colloquially referred to as "typos" and approximately 90% are unlikely to be repeated in normal spans of text.

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

“…The bulk of the errors there were shown to be single-character insertions, deletions and substitutions, in line with the findings of previous studies, the largest of which was [5]. In Table 3 and Table 4 we list comparable statistics obtained from the OCRed corpora we here work with: statistics on 5,047 mainly OCR-errors from the sgd and 3,799 from the ddd.…”

Revised N-Gram based Automatic Spelling Correction Tool to Improve Retrieval Effectiveness

“…Many approaches have been applied since people started to deal with this problem. Different techniques like edit distance [4], rule-based techniques [10], n-grams [20], probabilistic techniques [14], neural nets [15], similarity key techniques [16,17] and noisy channel model [18,19] have been proposed. All of these are based on the idea of calculating the similarity between the misspelled word and the words contained in a dictionary.…”

Misinformation and Bias in Metadata Processing: Matching in Large Databases

“…Pollock and Zamora report on a spelling error detection project at Chemical Abstracts Service (CAS) and charac terize the types of errors they found. 8 Chemical Abstracts databases are among the most searched databases in the world. CAS is usually characterized as a set of sources with considerable depth and breadth.…”