Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems

Abstract

With the advent of word representations, word similarity tasks are becoming increasing popular as an evaluation metric for the quality of the representations. In this paper, we present manually annotated monolingual word similarity datasets of six Indian languages -Urdu, Telugu, Marathi, Punjabi, Tamil and Gujarati. These languages are most spoken Indian languages worldwide after Hindi and Bengali. For the construction of these datasets, our approach relies on translation and re-annotation of word similarity datasets of English. We also present baseline scores for word representation models using state-of-the-art techniques for Urdu, Telugu and Marathi by evaluating them on newly created word similarity datasets.

IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

“…Datasets are available for some tasks for a few languages. The following are some of the prominent publicly available datasets 2 : word similarity (Akhtar et al, 2017), word analogy (Grave et al, 2018), text classification, sentiment analysis (Akhtar et al, 2016;Mukku and Mamidi, 2017), paraphrase detection (Anand , QA (Clark et al, 2020;, discourse mode classification (Dhanwal et al, 2020), etc.. We also create datasets for some tasks, most of which span all major Indian languages. We bun- dle together the existing datasets and our newly created datasets to create the IndicGLUE benchmark.…”

“…Given the morphological richness of Indian languages we train FastText word embeddings which are known to be more effective for such languages. To evaluate these embeddings we curate a benchmark comprising of word similarity and analogy tasks (Akhtar et al, 2017;Grave et al, 2018), text classification tasks, sentence classification tasks (Akhtar et al, 2016;Mukku and Mamidi, 2017), and bilingual lexicon induction tasks. On most tasks, the word embeddings trained on our IndicCorp outperform similar embeddings trained on existing corpora for Indian languages.…”

Word Similarity Datasets for Thai: Construction and Evaluation

“…For the translation from English into other languages, many researchers [2], [10], [11] use the strategy of involving two independent translators, and in case of disagreement on the translations, to have a third expert decide -we applied this approach, too, with translators being Thai academics who are fluent in English. As in Camacho-Collados et al [2], during translation, the annotators were presented the original similarity score of the word pair, in order to help selecting the correct translation for the intended meanings of the words.…”

“…Sophisticated methods to tackle the problem of OOV words are beyond the scope of this work, but we implemented a baseline method to address the issue using the deepcut tokenizer. 11 As word segmentation is a crucial step in NLP-processing of Thai text, there exist a number of tools and research papers on the topic. In the past, different dictionary-based word segmentation approaches, such as longest-matching and maximal matching, have been employed.…”

“…Some of the well-known existing word similarity datasets include RG-65 [6] (containing 65 word pairs), WordSim-353 [7] (353 pairs), and SimLex-999 [8] (with 999 word pairs). Typically, word similarity gold-standards were ini- VOLUME 4, 2016 tially created for the English language only, although in recent years there have been increased efforts to translate some of the datasets into various other European languages [1], [2], Chinese [9], Indian languages [10], etc. However, for Thai language, to the best of our knowledge, there only exists a very small dataset (65 word pairs) by Osathanunkul et al [11] based on the Rubenstein & Goodenough's RG-65 [6] dataset -which is too limited in size and other aspects like domain coverage to allow a comprehensive evaluation of Thai word embedding models.…”