Automatic n-gram language model creation from web resources

Nisimura, Ryuichi; Komatsu, Kumiko; Kuroda, Yuka; Nagatomo, Kentaro; Lee, Akinobu; Saruwatari, Hiroshi; Shikano, Kiyohiro

doi:10.21437/eurospeech.2001-501

Cited by 12 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classically, we can keep all the sentences exclusively made with words of the task-specific vocabulary preliminary defined. The other method proposed in [10] is a text filtering algorithm based on character perplexity. However, it needs a "Standard Language Model" for reference and there is not any such reference model available in minority languages.…”

Section: Sentence Filteringmentioning

confidence: 99%

Using the web for fast language model construction in minority languages

Le¹,

Bigi²,

Besacier³

et al. 2003

8th European Conference on Speech Communication and Technology (Eurospeech 2003)

View full text Add to dashboard Cite

The design and construction of a language model for minority languages is a hard task. By minority language, we mean a language with small available resources, especially for the statistical learning problem. In this paper, a new methodology for fast language model construction in minority languages is proposed. It is based on the use of Web resources to collect and make efficient textual corpora. By using efficient filtering techniques, this methodology allows a quick and efficient construction of a language model with a small cost in term of computational and human resources.Our primary experiments have shown excellent performance of the Web language models vs newspaper language models using the proposed filtering methods on a majority language (French). Following the same way for a minority language (Vietnamese), a valuable language model was constructed in 3 month with only 15% new development to convert some filtering tools.

show abstract

Section: Sentence Filteringmentioning

confidence: 99%