Using the web for fast language model construction in minority languages

Abstract

The design and construction of a language model for minority languages is a hard task. By minority language, we mean a language with small available resources, especially for the statistical learning problem. In this paper, a new methodology for fast language model construction in minority languages is proposed. It is based on the use of Web resources to collect and make efficient textual corpora. By using efficient filtering techniques, this methodology allows a quick and efficient construction of a language model with a small cost in term of computational and human resources.Our primary experiments have shown excellent performance of the Web language models vs newspaper language models using the proposed filtering methods on a majority language (French). Following the same way for a minority language (Vietnamese), a valuable language model was constructed in 3 month with only 15% new development to convert some filtering tools.

Digital tools, rhetoric, and meaning-making: A critical exploration of addressing the digital divide and accessibility to improve our digital landscape

“…Technology has been, and will continue to be, used to help spread linguistic knowledge and preserve indigenous languages. We have seen this occur in many fashions, such as the creation and use of dictionaries and corpora (Banea, Mihalcea, & Wiebe, 2008;Frawley, Hill, & Munro, 2002;Le, Bigi, Besacier, & Castelli, 2003), automatic speech recognition systems (Barnard, Davel, & van Heerden 2009;Barnard, Davel, van Huyssteen, 2010;Barnard, Davel, van Heerden, de Wet, and Badenhorst, 2014;Besacier et al, 2014;Le & Besacier, 2009), and natural language processing and language engineering 3 (Aduriz et al, 2006;Elliott, Glauert, Kennaway, Marshall, & Safar, 2008;Sarasola, 2000;Streiter, Scannell, & Stuflesser, 2006). All of these technologies either can and have been used towards the development of CALL tools or are already used as CALL tools in their current form.…”