Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit

Abstract

The configurational information in sentences of a free word order language such as Sanskrit is of limited use. Thus, the context of the entire sentence will be desirable even for basic processing tasks such as word segmentation. We propose a structured prediction framework that jointly solves the word segmentation and morphological tagging tasks in Sanskrit. We build an energy based model where we adopt approaches generally employed in graph based parsing techniques (McDonald et al., 2005a;Carreras, 2007). Our model outperforms the state of the art with an F-Score of 96.92 (percentage improvement of 7.06%) while using less than one tenth of the task-specific training data. We find that the use of a graph based approach instead of a traditional lattice-based sequential labelling approach leads to a percentage gain of 12.6% in F-Score for the segmentation task. 1

A "Verbal Thermometer" for Assessing Neurodegenerative Disease: Automated Measurement of Pronoun and Verb Ratio from Speech

“…More contextual features and different tagger architectures have been found to be much better for tagging languages that lack articles like the and languages with freer word order than English, e.g. Sanskrit [28] or Turkish [29]. For language of people with neurological deficits, similar such alternative taggers these might classify parts of speech more like a human expert than the present system does.…”

A Graph-Based Framework for Structured Prediction Tasks in Sanskrit

“…In MRLs, hand-crafted features still form a crucial component in contributing to the performance of the state-of-the-art systems for tasks such as morphological parsing and dependency parsing More et al, 2019;Seeker and Ç etinoglu 2015). But Krishna et al (2018) learn a feature function using the Path Ranking Algorithm (PRA) (Lao and Cohen 2010) for the joint task of word segmentation and morphological parsing. PRA essentially maps the problem of learning a feature function to that of automatic learning of horn clauses (Gardner, Talukdar, and Mitchell 2015), where each clause is a morphological constraint.…”

“…The domain knowledge required here confines to just defining the literals, the combinations of which will be used to form the clauses. In Krishna et al (2018), morphological tags and grammatical categories form the literals and the feature (clause) values are calculated using distributional information from a morphologically tagged corpus. We find that the same feature function can be used effectively for all the standalone and joint tasks we experimented with, including the downstream morphosyntactic tasks.…”

“…However, the sentence will still maintain the same syntactic analysis, irrespective of its varying word orders (Scharf et al 2015;Gillon and Shaer 2005). Recently, Krishna et al (2018) have shown that approaches for non-sequential processing of Sanskrit sentences result in better system performance even for lowlevel tasks such as word-segmentation and morphological parsing. In this work, we extend the energy-based model (EBM) for joint modeling of word segmentation and morphological parsing proposed by Krishna et al (2018) into a general graph-based parsing framework for multiple structured prediction tasks in Sanskrit.…”

“…Recently, Krishna et al (2018) have shown that approaches for non-sequential processing of Sanskrit sentences result in better system performance even for lowlevel tasks such as word-segmentation and morphological parsing. In this work, we extend the energy-based model (EBM) for joint modeling of word segmentation and morphological parsing proposed by Krishna et al (2018) into a general graph-based parsing framework for multiple structured prediction tasks in Sanskrit. We extend the framework to include two downstream syntax-level tasks, dependency parsing and syntactic linearization.…”