A language and character set determination method based on N-gram statistics

Abstract

An N-gram-based language, script, and encoding scheme-detection method is introduced in this article. The method detects language, script, and encoding schemes using a target text document encoded by computer by checking how many byte sequences of the target match the byte sequences that can appear in the texts belonging to a language, script, and encoding scheme. This detection mechanism is different from conventional N-gram-based methods in that its threshold for any category is uniquely predetermined. The method was originally created for a survey of web pages conducted to find how many web pages are written in a particular language, script, and encoding scheme. The requirement is that the method must be able to respond to either "correct answer" or "unable to detect" where "unable to detect" includes "other than registered." There are some minor problems with this method, but its effectiveness as a language, script, and encoding scheme-detection method has been confirmed by experiments.

Monotone Increasing Binary Similarity and Its Application to Automatic Document-Acquisition of a Category

“…The authors introduced a binary indexed linear classifier [1] for language identification in the Language Observatory Project. The term weighting schemes of this classifier are configured as follows:…”

“…One information retrieval technique, a method that finds documents that are "similar" to a given snippet of text will make searching more efficient in many ways. The authors have introduced binary similarity [1]: a linear classifier in which training documents are indexed in a binary manner. In this indexing system, the weight of a term is set to 1 if it occurs in the training documents and 0 if it does not, as defined in Eq.…”

“…In this case, a new category must be registered manually, and then training documents must be collected enough to learn about the category. The classifier is employed because it performs well without the need for stop-word reduction, which is almost impossible for a newly registered category [1].…”

“…The process of finding similar documents is performed in a manner that repeatedly expands the set of training documents by adding the most similar documents. The advantage of the monotone increasing property is that a) the threshold value is decided by the regulation "a number close to 1, but less than 1" [1] and b) the monotone increasing property induces the correct timing to stop the expanding process, which will be discussed in Sect. 5.2.…”

Orthographic Reforms of Standard Malay Online: Towards Better Pronunciation and Construction of a Cross-language Environment

“…The huge data are stored in clusters of 40 servers at Nagaoka University of Technology under the management of the Language Observatory (LO) project (Mikami et al 2005). The system based on N-gram technology (Suzuki et al 2002) has been extensively used in the past four years in a project for LSE identification of huge domains.…”