Brahmi-Net: A transliteration and script conversion system for languages of the Indian subcontinent

Abstract

We present Brahmi-Net -an online system for transliteration and script conversion for all major Indian language pairs (306 pairs). The system covers 13 Indo-Aryan languages, 4 Dravidian languages and English. For training the transliteration systems, we mined parallel transliteration corpora from parallel translation corpora using an unsupervised method and trained statistical transliteration systems using the mined corpora. Languages which do not have parallel corpora are supported by transliteration through a bridge language. Our script conversion system supports conversion between all Brahmi-derived scripts as well as ITRANS romanization scheme. For this, we leverage co-ordinated Unicode ranges between Indic scripts and use an extended ITRANS encoding for transliterating between English and Indic scripts. The system also provides top-k transliterations and simultaneous transliteration into multiple output languages. We provide a Python as well as REST API to access these services. The API and the mined transliteration corpus are made available for research use under an open source license.

Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning

“…We ran Stage 2 for 5 iterations. For a rule-based baseline, we used the script conversion method implemented in the Indic NLP Library 2 (Kunchukuttan et al, 2015) which is based on phonemic correspondences.…”

“…Comparison with supervised system and some resource-constrained approaches: We compared our best substring-based model (based on sim1 prior) with a supervised system and the following resource-constrained transliteration systems built using: (i) Mined pairs from a translation corpus: We experimented with bn-hi on the Brahminet mined pairs corpus (Kunchukuttan et al, 2015). Mined corpora involving kn were not available.…”

Judicious Selection of Training Data in Assisting Language for Multilingual Neural NER

“…We convert Tamil, Bengali and Malayalam data to the Devanagari script using the Indic NLP li-2 Data is available here: http://www.cfilt.iitb. ac.in/ner/annotated_corpus/ brary 3 (Kunchukuttan et al, 2015) thereby, allowing sharing of sub-word features across the Indian languages. For Indian languages, the annotated data followed the IOB format.…”

Speech Synthesis for Mixed-Language Navigation Instructions

“…Brahmi-Net transliteration [11] considers this problem similar to a phrase based translation problem, through which sequences of characters from source to the target language are learnt, where the parallel corpus is trained using Moses. This system supports 13 Indo-Aryan languages, 4 Dravidian languages and English including 306 language pairs for statistical transliteration.…”