In Natural Language Processing (NLP) pipelines, Named Entity Recognition (NER) is one of the preliminary problems, which marks proper nouns and other named entities such as Location, Person, Organization, Disease etc. Such entities, without an NER module, adversely affect the performance of a machine translation system. NER helps in overcoming this problem by recognising and handling such entities separately, although it can be useful in Information Extraction systems also. Bhojpuri, Maithili and Magahi are low resource languages, usually known as Purvanchal languages. This paper focuses on the development of an NER benchmark dataset for Machine Translation systems developed to translate from these languages to Hindi by annotating parts of the available corpora with named entities. Bhojpuri, Maithili and Magahi corpora of sizes 228373, 157468 and 56190 tokens, respectively, were annotated using 22 entity labels. The annotation considers coarse-grained annotation labels followed by the tagset used in one of the Hindi NER datasets. We also report a Deep Learning baseline that uses an LSTM-CNNs-CRF model. The lower baseline F 1 -scores from the NER tool obtained by using Conditional Random Fields models are 70.56% for Bhojpuri, 73.19% for Maithili and 84.18% for Magahi. The Deep Learning-based technique (LSTM-CNNs-CRF) achieved 61.41% for Bhojpuri, 71.38% for Maithili and 86.39% for Magahi. As the results show, LSTM-CNNs-CRF fails to outperform the lower baseline in the case of Bhojpuri and Maithili, which have more data in terms of the number of tokens, but not in terms of the number of named entities. However, the cross-lingual model training of LSTM-CNNs-CRF for Bhojpuri and Maithili performed better than the CRF.

The sentiment of a word varies based on its context of usage: the words used around it and the part-of-speech it is used as. This paper proposes a technique to suggest the sentiment of a word by combining its part-of-speech and the semantic similarities of its co-occurrences with both context-specific and pre-trained embeddings to achieve powerful and fast results. A study was conducted across domains and sub-domains to measure variance of sentiment by switching domains and switching context within the same domain. Re-scoring a commonly used polarity lexicon showed that 10% of words changed scores while switching domains and 8% changed scores within domains while switching context. Part of Speech analysis on 65,353 commonly used sentiment lexicons showed that 81% of sentiment bearing (non-neutral) lexicons were of the tags NN (Common Noun), JJ (Adjective) or NNS (Proper Noun).