A New Word Language Model Evaluation Metric for Character Based Languages

Wang, Peilu; Sun, Ruihua; Zhang, Hai; Yu, Kai

doi:10.1007/978-3-642-41491-6_29

Cited by 9 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For character-based languages (e.g. Chinese), CER is commonly used instead of WER as the measure for OCR, and, thus, we report only the CER [66]. These error values clearly state that Character Degradation is the effect that affects the transcription of the documents the most.…”

Section: Daniel-sysmentioning

confidence: 99%

Assessing the impact of OCR noise on multilingual event detection over digitised documents

et al. 2022

View full text Add to dashboard Cite

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

Section: Daniel-sysmentioning

confidence: 99%

Assessing the impact of OCR noise on multilingual event detection over digitised documents

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The developed language model is measured using a perplexity score since it will be used in an ASR system. As described in [20], [21], perplexity is a commonly used metric to measure the performance of a word-based language model applied in an ASR model. This metric has two advantages.…”

Section: Collapsed Gibbs Samplingmentioning

confidence: 99%

“…Firstly, it is calculated independently with no real ASR. It is categorized as an intrinsic evaluation that is much simpler than an extrinsic one by evaluating the language model on the real ASR model [20]. Secondly, it has a high correlation with word error rate (WER) in an ASR, especially when the models are trained using the same training set of data.…”

Section: Collapsed Gibbs Samplingmentioning

confidence: 99%

Developing an Adaptive Language Model for Bahasa Indonesia

Hidayatullah¹,

Suyanto²

2019

ijacsa

View full text Add to dashboard Cite

A language model is one of the important components in a speech recognition system. It is commonly developed using a statistical method called n-gram. However, a standard n-gram cannot be used for general domains with so many ambiguous semantics of sentences. This paper focuses on developing an adaptive n-gram language model for Bahasa Indonesia. First, a text corpus of ten million distinct sentences is crawled from hundreds of websites of news, magazines, personal blogs, and writing forums. The text corpus is then used to construct an adaptive language model using Latent Dirichlet Allocation (LDA) with Collapsed Gibbs Sampling (CGS) training method. Compare to the standard n-gram, the adaptive language model gives a better performance in the word selection to produce the best sentence.

show abstract

“…The Chinese part of the corpus is segmented into words before LM training. Maximum matching word segmentation is used with a large word vocabulary V extracted from web data provided by (Wang et al, 2013b). The pinyin part is segmented according to the Chinese part.…”

Section: Corpora Tools and Experiments Settingsmentioning

confidence: 99%

A Joint Graph Model for Pinyin-to-Chinese Conversion with Typo Correction

Jia¹,

Zhao

2014

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

It is very import for Chinese language processing with the aid of an efficient input method engine (IME), of which pinyinto-Chinese (PTC) conversion is the core part. Meanwhile, though typos are inevitable during user pinyin inputting, existing IMEs paid little attention to such big inconvenience. In this paper, motivated by a key equivalence of two decoding algorithms, we propose a joint graph model to globally optimize PTC and typo correction for IME. The evaluation results show that the proposed method outperforms both existing academic and commercial IMEs.

show abstract

A New Word Language Model Evaluation Metric for Character Based Languages

Cited by 9 publications

References 5 publications

Assessing the impact of OCR noise on multilingual event detection over digitised documents

Assessing the impact of OCR noise on multilingual event detection over digitised documents

Developing an Adaptive Language Model for Bahasa Indonesia

A Joint Graph Model for Pinyin-to-Chinese Conversion with Typo Correction

Contact Info

Product

Resources

About