Development of Bangla Spell and Grammar Checkers: Resource Creation and Evaluation

Abstract

A spell and grammar checker is profoundly essential for diverse publications especially for Bangla language in particular as it is spoken by millions of native speakers around the world. Considering the lack of research efforts, we demonstrate the development of a comprehensive Bangla spell and grammar checker with necessary resources. At first, a full-fledged and generalised Bangla monolingual corpus comprising over 100 million words has been built by scraping reputed, diversified online sources and then an extensive Bangla lexicon consisting of over 1 million unique words has been extracted from that corpus. Based on these corpus and lexicon, we have developed a combined spell and grammar checker application that simultaneously detects distinct spelling and grammatical mistakes and provides appropriate suggestions for both as well. The spell checker uses the Double Metaphone algorithm and Edit distance based on the distributed lexicons and numerical suffix dataset to detect all types of Bangla spelling mistakes with an accuracy rate of 97.21% individually. The grammar checker detects errors based on language model probability i.e. combination of bigram and trigram, and generates suggestions based on the Cosine similarity measure with the accuracy rate of 94.29% individually. The datasets and codes used in this work are freely available at https://git.io/JzJ4w.

An improved extrinsic monolingual plagiarism detection approach of the Bengali text

“…This process took more than 1 month. Moreover, we have taken nearly 2 million sentences from the reputed Bengali monolingual corpus NHMono01 [26] in the newspaper domain. The complete newspaper dataset formation process is shown in Figure 4.…”

Inclusive bidirectional conversion system between Chittagonian and standard Bangla

“…The double metaphone encoding table [16], [17] has been implemented as the encoder. Based on the probable phonetics of the expression, each Bangla letter has been encoded with one or more English letters.…”

Compilation, Analysis and Application of a Comprehensive Bangla Corpus KUMono

“…In recent years, a few Bangla language researchers have demonstrated a keen interest in Bangla corpus development and Bangla text analysis. Some notable progress has been documented in corpus creation in [15], [16], [17], [18], [19], [20] and knowledge engineering on Bangla language in [21], [22], [23], [24], etc. The first electronic Bangla corpus was constructed by the Central Institute of Indian Languages (CIIL) from 1991 to 1995 [25].…”

“…They have also presented several results of the study of features of the Bangla language based on empirical analysis of the corpus. In [20] a new corpus NHMono01 consisting of 100,142,522 tokens was developed. Their developed spell and grammar checker application of the corpus demonstrated some excellent results.…”