Formal Methods of Tokenization for Part-of-Speech Tagging

Graña, Jorge; Barcala, Francisco-Mario; Vilares, Jesús

doi:10.1007/3-540-45715-1_22

Cited by 14 publications

(10 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pre-processing One of the most important prior tasks for robust part-ofspeech tagging is the correct tokenization or segmentation of the texts [56]. Arabic language has rich and complex morphology [44], [57], [58].…”

Section: A the Proposed System Architecturementioning

confidence: 99%

Classifiers combination to arabic morphosyntactic disambiguation

Albared

Omar

Aziz

2009

2009 International Conference on Electrical Engineering and Informatics

View full text Add to dashboard Cite

Parts of speech tagging forms the important preprocessing step in many of the natural language processing applications like text summarization, question answering and information retrieval system. MorphoSyntactic disambiguation (part of speech tagging) is the process of classifying every word in a given context to its appropriate part of speech. In this paper, we first review all the supervised machine learning approaches that have been used in the part of speech tagging. Then we review all the Arabic works to compare and to confirm our need to develop an accurate and efficient Arabic MorphoSyntactic Disambiguation system. Finally we propose a classifiers combination experimental framework for Arabic part of speech tagger in which three diverse probabilistic classifiers (Hidden Markov, Maximum Entropy and Transformation Based Learning) are combined using many different combination strategies to exploit their advantages Keywords-natural language processing, MorphoSyntactic disambiguation, machine learning.

show abstract

Section: A the Proposed System Architecturementioning

confidence: 99%

Classifiers combination to arabic morphosyntactic disambiguation

Albared

Omar

Aziz

2009

2009 International Conference on Electrical Engineering and Informatics

View full text Add to dashboard Cite

show abstract

“…The problems of Spanish pre processing and segmentation have been studied in depth by Graña et al (2002). Their work presents a linguistically based pre processing segmenter system able to deal successfully with complex phenomena, such as multiword expressions, contractions, enclitic pronouns attached to verbs, and even segmentation ambiguities.…”

Section: Pre Processing and Text Segmentationmentioning

confidence: 99%

“…These tools need to take into account the inflectional morphology of each specific language, their irregularities and even their segmentation characteristics. This is the case, for example, of the work developed for both Spanish and Galician with MrTagoo tagger lemmatizer (Graña et al 2001;Graña et al 2002). Further, the forms of the user queries should be studied in non English languages to realize how users type their queries; whether, for example, they use the same terms in various inclinations with various endings.…”

Section: Conflationmentioning

confidence: 99%

Current research issues and trends in non-English Web searching

et al. 2009

Self Cite

View full text Add to dashboard Cite

With increasingly higher numbers of non English language web searchers the problems of efficient handling of non English Web documents and user queries are becoming major issues for search engines. The main aim of this review paper 1 is to make researchers aware of the existing problems in monolingual non English Web retrieval by providing an overview of open issues. A significant number of papers are reviewed and the research issues investigated in these studies are categorized in order to identify the research questions and solutions proposed in these papers. Further research is proposed at the end of each section.

show abstract

“…This working hypothesis is not realistic due to the heterogeneous nature of the application texts and their sources. For this reason, we have developed a preprocessor module [4,1], an advanced tokenizer which performs the following tasks:…”

Section: The Preprocessormentioning

confidence: 99%

Practical NLP-Based Text Indexing

Vilares

Barcala

Alonso

et al. 2002

Advances in Artificial Intelligence — IBERAMIA 2002

Self Cite

View full text Add to dashboard Cite

Abstract. We consider a set of natural language processing techniques based on finite-state technology that can be used to analyze huge amounts of texts. These techniques include an advanced tokenizer, a part-of-speech tagger that can manage ambiguous streams of words, a system for conflating words by means of derivational mechanisms, and a shallow parser to extract syntactic-dependency pairs. We propose to use these techniques in order to improve the performance of standard indexing engines.

show abstract

Formal Methods of Tokenization for Part-of-Speech Tagging

Cited by 14 publications

References 2 publications

Classifiers combination to arabic morphosyntactic disambiguation

Classifiers combination to arabic morphosyntactic disambiguation

Current research issues and trends in non-English Web searching

Practical NLP-Based Text Indexing

Contact Info

Product

Resources

About