Development and Evaluation of a Korean Treebank and its Application to NLP

Han, Chung–hye; Han, Na-Rae; Ko, Eon-Suk; Palmer, Martha

doi:10.29403/li.6.1.7

Cited by 17 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…How can we estimate the rate at which VPs are transitive as opposed to intransitive? Looking in the Penn Korean Treebank (KTB) (Han et al, 2002) we identify particular structural configurations that the annotators used to flag these alternatives. 6 These configurations are schematically illustrated below in 7a and 7b.…”

Section: Counting Constructions In Corporamentioning

confidence: 99%

“…Korean Treebank 2.0 (Han et al 2002) was used to obtain an estimation of the frequencies of the relevant structures in Korean. The texts of the corpus are a selection of Korean Press Agency news articles in 2000, consisting of 5,010 sentences and 132,040 words.…”

Section: Appendix 1: Attestation Counts (A) Koreanmentioning

confidence: 99%

“…Probability distributions over this remainder set characterize the expectations that a comprehender would have, partway through the sentence. To estimate them, we weight the grammars' structural rules in accordance with corpora like the Penn Chinese Treebank 7 (Xue et al 2005), the Kyoto Corpus 4 (Kurohashi and Nagao 2003) and the Penn Korean Treebank 2 (Han et al 2002). The uncertainty or entropy of this remainder set quantifies the notion of sentence-medial ambiguity.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Uncertainty in processing relative clauses across East Asian languages

Yun

Chen

Hunter

et al. 2014

J East Asian Linguist

View full text Add to dashboard Cite

The processing difficulty profile for relative clauses in Chinese, Japanese and Korean represents a challenge for theories of human parsing. We address this challenge using a grammar-based complexity metric, one that reflects a minimalist analysis of relative clauses for all three languages as well as structure-dependent corpus distributions. Together, these define a comprehender's degree of uncertainty at each point in a sentence. We use this idea to quantify the intuition that people do comprehension work as they incrementally resolve ambiguity, word by word. We find that downward changes to this quantitative measure of uncertainty derive observed processing contrasts between Subject-and Object-extracted relative clauses. This demonstrates that the complexity metric, in conjunction with a minimalist grammar and corpus-based weights, accounts for the widely-observed Subject Advantage. Electronic supplementary material The online version of this article

show abstract

Section: Counting Constructions In Corporamentioning

confidence: 99%

Section: Appendix 1: Attestation Counts (A) Koreanmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Uncertainty in processing relative clauses across East Asian languages

Yun

Chen

Hunter

et al. 2014

J East Asian Linguist

View full text Add to dashboard Cite

show abstract

“…The Korean treebank (KTB) was developed and evaluated in Han et al (2002). Korean is an agglutinative language with a very productive inflectional system.…”

Section: Treebank Constructionmentioning

confidence: 99%

Vietnamese treebank construction and entropy-based error detection

Nguyen

et al. 2015

Lang Resources & Evaluation

View full text Add to dashboard Cite

Treebanks, especially the Penn treebank for natural language processing (NLP) in English, play an essential role in both research into and the application of NLP. However, many languages still lack treebanks and building a treebank can be very complicated and difficult. This work has a twofold objective. Firstly, to share our results in constructing a large Vietnamese treebank (VTB) with three levels of annotation including word segmentation, part-of-speech tagging, and syntactic analysis. Major steps in the treebank construction process are described with particular regard to specific Vietnamese properties such as lack of word delimiter and isolation. Those properties make sentences highly syntactically ambiguous, and therefore it is difficult to ensure a high level of agreement among annotators. Various studies of Vietnamese syntax were employed not only to define annotations but also to systematically deal with ambiguities. Annotators were supported by automatic labelling tools, which are based on statistical machine learning methods, for sentence pre-processing and a tree editor for supporting manual annotation. As a result, an annotation agreement of around 90 % was achieved. Our second objective is to present our method for automatically finding errors and inconsistencies in treebank corpora and its application to the construction of the VTB. This method employs the Shannon entropy measure in a manner that the more reduced entropy the more corrected errors in a treebank. The method ranks error candidates by using a scoring function based on conditional entropy. Our experiments showed that this method detected high-error-density subsets of original error candidate sets, and that the corpus entropy was significantly reduced after error correction. The size of these subsets was only about one third of the whole set, while these subsets contained 80-90 % of the total errors. This method can also be applied to languages similar to Vietnamese.

show abstract

“…Arabic is also gaining significant attention in the research community and an Arabic Treebank [Maamouri et al 2004] consisting of part of speech and morphological annotations is available. The Linguistic Data Consortium has also released the Korean Treebank [Han et al 2002]. An excellent compilation of resources is available at this Web site.…”

Section: Introductionmentioning

confidence: 99%

An Information-Extraction System for Urdu---A Resource-Poor Language

Mukund

Srihari

Peterson³

2010

ACM Transactions on Asian Language Information Processing

View full text Add to dashboard Cite

There has been an increase in the amount of multilingual text on the Internet due to the proliferation of news sources and blogs. The Urdu language, in particular, has experienced explosive growth on the Web. Text mining for information discovery, which includes tasks such as identifying topics, relationships and events, and sentiment analysis, requires sophisticated natural language processing (NLP). NLP systems begin with modules such as word segmentation, part-of-speech tagging, and morphological analysis and progress to modules such as shallow parsing and named entity tagging. While there have been considerable advances in developing such comprehensive NLP systems for English, the work for Urdu is still in its infancy. The tasks of interest in Urdu NLP includes analyzing data sources such as blogs and comments to news articles to provide insight into social and human behavior. All of this requires a robust NLP system. The objective of this work is to develop an NLP infrastructure for Urdu that is customizable and capable of providing basic analysis on which more advanced information extraction tools can be built. This system assimilates resources from various online sources to facilitate improved named entity tagging and Urdu-to-English transliteration. The annotated data required to train the learning models used here is acquired by standardizing the currently limited resources available for Urdu. Techniques such as bootstrap learning and resource sharing from a syntactically similar language, Hindi, are explored to augment the available annotated Urdu data. Each of the new Urdu text processing modules has been integrated into a general text-mining platform. The evaluations performed demonstrate that the accuracies have either met or exceeded the state of the art.

show abstract

Development and Evaluation of a Korean Treebank and its Application to NLP

Cited by 17 publications

References 0 publications

Uncertainty in processing relative clauses across East Asian languages

Uncertainty in processing relative clauses across East Asian languages

Vietnamese treebank construction and entropy-based error detection

An Information-Extraction System for Urdu---A Resource-Poor Language

Contact Info

Product

Resources

About