Unsupervised segmentation of Chinese text by use of branching entropy

Jin, Zhihui; Tanaka-Ishii, Kumiko

doi:10.3115/1273073.1273129

Cited by 39 publications

(38 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…따라서 이러한 공공 및 비즈 니스 목적에 부합하는 새로운 가치를 창출하기 위한 텍스트 데이터 가공 및 분석 방법론의 개발을 주로 다루는 텍스트마 이닝(text mining)에 대한 중요성이 크게 부각되고 있는 실정이 다. 텍스트마이닝의 주요 활용 분야로는 키워드 기반의 연관 관계 분석(keyword-based association analysis), 자동 문서 분류 (automatic document classification), 문서간 유사도 탐색(similarity detection between documents), 특이 문서 탐지(anomaly detection), 문서 간 링크 분석(link analysis) 등이 있다 (Hotho et al, 단어 인식 방법은 학습데이터를 이용하여 단어를 추정할 수 있는 정보를 학습하는 지도학습 기반 방법과 사전 지식 없이 통계적인 정보를 기반으로 단어를 추정하는 비지도학습 기반 방법으로 나눌 수 있다 (Jin and Tanaka-Ishii, 2006;Zhao and Kit, 2007). (Sun et al, 1998;Feng et al, 2004;Jin, 2006).…”

Section: 서 론unclassified

KR-WordRank : An Unsupervised Korean Word Extraction Method Based on WordRank

Kim¹,

Cho

Kang³

2014

Journal of Korean Institute of Industrial Engineers

View full text Add to dashboard Cite

A Word is the smallest unit for text analysis, and the premise behind most text-mining algorithms is that the words in given documents can be perfectly recognized. However, the newly coined words, spelling and spacing errors, and domain adaptation problems make it difficult to recognize words correctly. To make matters worse, obtaining a sufficient amount of training data that can be used in any situation is not only unrealistic but also inefficient. Therefore, an automatical word extraction method which does not require a training process is desperately needed. WordRank, the most widely used unsupervised word extraction algorithm for Chinese and Japanese, shows a poor word extraction performance in Korean due to different language structures. In this paper, we first discuss why WordRank has a poor performance in Korean, and propose a customized WordRank algorithm for Korean, named KR-WordRank, by considering its linguistic characteristics and by improving the robustness to noise in text documents. Experiment results show that the performance of KR-WordRank is significantly better than that of the original WordRank in Korean. In addition, it is found that not only can our proposed algorithm extract proper words but also identify candidate keywords for an effective document summarization. †

show abstract

Section: 서 론unclassified

KR-WordRank : An Unsupervised Korean Word Extraction Method Based on WordRank

Kim¹,

Cho

Kang³

2014

Journal of Korean Institute of Industrial Engineers

View full text Add to dashboard Cite

show abstract

“…Many works have investigated the use of such free data to address the problems for the resource-poor domains [Jin and Tanaka-Ishii, 2006;Zhao and Kit, 2008;Li and Sun, 2009;Sun and Xu, 2011;Zhang et al, 2013]. Meanwhile, from web pages such as Wikipedia, a sentence's partial segmentation information can be inferred from hyperlinks, which produces partially-labeled data.…”

Section: Introductionmentioning

confidence: 99%

Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

Zhao¹,

Zhang²,

Wang³

et al. 2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Most existing Chinese word segmentation (CWS) methods are usually supervised. Hence, large-scale annotated domain-specific datasets are needed for training. In this paper, we seek to address the problem of CWS for the resource-poor domains that lack annotated data. A novel neural network model is proposed to incorporate unlabeled and partiallylabeled data. To make use of unlabeled data, we combine a bidirectional LSTM segmentation model with two character-level language models using a gate mechanism. These language models can capture co-occurrence information. To make use of partially-labeled data, we modify the original cross entropy loss function of RNN. Experimental results demonstrate that the method performs well on CWS tasks in a series of domains.

show abstract

“…Another notable work is nVBE: Magistry and Sagot (2012) proposed a model based on the Variation of Branching Entropy. By adding normalization and viterbi decoding, they improve performance over Jin and Tanaka-Ishii (2006) and remove most of the parameters and thresholds from the model.…”

Section: Related Workmentioning

confidence: 99%

“…Zhao and Kit (2008) compared several popular unsupervised models within a unified framework. They tried various types of goodness measures, such as Description Length Gain (DLG) proposed by Kit and Wilks (1999), Accessor Variety (AV) proposed by and Boundary Entropy (Jin and Tanaka-Ishii, 2006). A notable goodness-based method is ESA: "Evaluation, Selection, Adjustment", which is proposed by Wang et al (2011) for unsupervised Mandarin Chinese word segmentation.…”

Section: Related Workmentioning

confidence: 99%

“…The first type uses carefully designed goodness measure to identify word candidates. Popular goodness measures include description length gain (DLG) (Kit and Wilks, 1999), accessor variety (AV) , boundary entropy (BE) (Jin and Tanaka-Ishii, 2006) and normalized variation of branching entropy (nVBE) (Magistry and Sagot, 2012) etc. Goodness measure based model is not segmentation model in a very strict meaning and is actually strong in generating word list without supervision.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Joint Model for Unsupervised Chinese Word Segmentation

Chen¹,

Chang²,

Pei³

2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

In this paper, we propose a joint model for unsupervised Chinese word segmentation (CWS). Inspired by the "products of experts" idea, our joint model firstly combines two generative models, which are word-based hierarchical Dirichlet process model and character-based hidden Markov model, by simply multiplying their probabilities together. Gibbs sampling is used for model inference. In order to further combine the strength of goodness-based model, we then integrated nVBE into our joint model by using it to initializing the Gibbs sampler. We conduct our experiments on PKU and MSRA datasets provided by the second SIGHAN bakeoff. Test results on these two datasets show that the joint model achieves much better results than all of its component models. Statistical significance tests also show that it is significantly better than stateof-the-art systems, achieving the highest F-scores. Finally, analysis indicates that compared with nVBE and HDP, the joint model has a stronger ability to solve both combinational and overlapping ambiguities in Chinese word segmentation.

show abstract

Unsupervised segmentation of Chinese text by use of branching entropy

Cited by 39 publications

References 7 publications

KR-WordRank : An Unsupervised Korean Word Extraction Method Based on WordRank

KR-WordRank : An Unsupervised Korean Word Extraction Method Based on WordRank

Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

A Joint Model for Unsupervised Chinese Word Segmentation

Contact Info

Product

Resources

About