Word Sense Disambiguation in Hindi Language Using Score Based Modified Lesk Algorithm

Tripathi, Praffullit; Mukherjee, Prasenjit; Hendre, Manik; Godse, Manish; Chakraborty, Basabi

doi:10.12785/ijcds/100185

Cited by 7 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Individual words are represented as real-valued vectors in a predefined vector space, where each word is mapped to a single vector; Words with identical meanings are represented similarly. This method was applied to many languages, like Hindi [18] and Arabic [19].…”

Section: Word and Sentence Embeddingmentioning

confidence: 99%

Exploring Sentence Embedding Representation For Arabic Question Answering

Lahbari,

Ouatik El Alaoui

2024

IJCDS

View full text Add to dashboard Cite

Question Answering Systems (QAS) are made to automatically provide accurate response to user questions that are phrased in natural language. Most of the existing QAS adopting traditional representations like word embedding and bag-of-words, have shown promising results. However, only a few works take into account the contextual information and meaning within texts to extract answers from huge sources of information. Moreover, dealing with Arabic open-domain question-answering systems is still challenging due to its rich morphology and ambiguity of words. To address these limitations, we introduce, in this study, a novel QA approach for the Arabic language that is based on passage retrieval and Sentence Embedding (SE) representation. It consists of three steps: (1) Question classification and query formulation, (2) Documents and passages retrieval, and (3) Answers extraction. In this work, we make use of the AraBERT transformer model to compute vector representation. This allows for considering both implicit semantics and the words' context within the text. Furthermore, in order to collect potential passages for user questions, we investigate a method for retrieving Arabic passages using the BM25 model, a query expansion process, and SE representation. The final answer is generated by fine-tuning AraBERT parameters and ranking passages so that the most relevant ones can be extracted. To assess our approach, we carried out several experiments on CLEF and TREC datasets using two different taxonomies. The obtained results show that the proposed method achieves 92% in terms of F1-score.

show abstract

Section: Word and Sentence Embeddingmentioning

confidence: 99%

Exploring Sentence Embedding Representation For Arabic Question Answering

Lahbari,

Ouatik El Alaoui

2024

IJCDS

View full text Add to dashboard Cite

show abstract

“…Tripathi, et al [14] used Lesk to disambiguate Hindi words. The appropriate sense of the polysemous word is determined through a scoring method that assigns a sense score to each token of the Hindi sentence.…”

Section: Related Workmentioning

confidence: 99%

Integrating Lesk Algorithm with Cosine Semantic Similarity to Resolve Polysemy for Setswana Language

Moape,

Olugbara,

Ojo

2024

IJACSA

View full text Add to dashboard Cite

Word Sense Disambiguation (WSD) serves as an intermediate task for enhancing text understanding in Natural Language Processing (NLP) applications, including machine translation, information retrieval, and text summarization. Its role is to enhance the effectiveness and efficiency of these applications by ensuring the accurate selection of the appropriate sense for polysemous words in diverse contexts. This task is recognized as an AI-complete problem, indicating its longstanding complexity since the 1950s. One of the earliest proposed solutions to address polysemy in NLP is the Lesk algorithm, which has seen various adaptations by researchers for different languages over the years. This study proposes a simplified, Lesk-based algorithm to resolve polysemy for Setswana. Instead of combinatorial comparisons among candidate senses that Lesk is based on that cause computational complexity, this study models word sense glosses using Bidirectional Encoder Representations from Transformers (BERT) and Cosine similarity measure, which have been proven to achieve optimal performance in WSD. The proposed algorithm was evaluated on Setswana and obtained an accuracy of 86.66 and an error rate of 14.34, surpassing the accuracy of other Lesk-based algorithms for other languages.

show abstract

“…The overall accuracy achieved is 63.71%. Tripathi et al [55] have used a Lesk algorithm along with a novel scoring method.…”

Section: Knowledge-based Techniquesmentioning

confidence: 99%

A Literature Survey on Word Sense Disambiguation for the Hindi Language

Gujjar,

Mago,

Kumari

et al. 2023

Information

View full text Add to dashboard Cite

Word sense disambiguation (WSD) is a process used to determine the most appropriate meaning of a word in a given contextual framework, particularly when the word is ambiguous. While WSD has been extensively studied for English, it remains a challenging problem for resource-scarce languages such as Hindi. Therefore, it is crucial to address ambiguity in Hindi to effectively and efficiently utilize it on the web for various applications such as machine translation, information retrieval, etc. The rich linguistic structure of Hindi, characterized by complex morphological variations and syntactic nuances, presents unique challenges in accurately determining the intended sense of a word within a given context. This review paper presents an overview of different approaches employed to resolve the ambiguity of Hindi words, including supervised, unsupervised, and knowledge-based methods. Additionally, the paper discusses applications, identifies open problems, presents conclusions, and suggests future research directions.

show abstract

Word Sense Disambiguation in Hindi Language Using Score Based Modified Lesk Algorithm

Cited by 7 publications

References 26 publications

Exploring Sentence Embedding Representation For Arabic Question Answering

Exploring Sentence Embedding Representation For Arabic Question Answering

Integrating Lesk Algorithm with Cosine Semantic Similarity to Resolve Polysemy for Setswana Language

A Literature Survey on Word Sense Disambiguation for the Hindi Language

Contact Info

Product

Resources

About