Multi-Grained Chinese Word Segmentation

Abstract

Traditionally, word segmentation (WS) adopts the single-granularity formalism, where a sentence corresponds to a single word sequence. However, Sproat et al. (1996) show that the inter-nativespeaker consistency ratio over Chinese word boundaries is only 76%, indicating single-grained WS (SWS) imposes unnecessary challenges on both manual annotation and statistical modeling. Moreover, WS results of different granularities can be complementary and beneficial for high-level applications.This work proposes and addresses multi-grained WS (MWS). First, we build a large-scale pseudo MWS dataset for model training and tuning by leveraging the annotation heterogeneity of three SWS datasets. Then we manually annotate 1,500 test sentences with true MWS annotations. Finally, we propose three benchmark approaches by casting MWS as constituent parsing and sequence labeling. Experiments and analysis lead to many interesting findings.

Multi-grained Chinese Word Segmentation with Weakly Labeled Data

“…Given an input sentence, the task of MWS is to retrieve all words of different granularities, which can be naturally organized as a hierarchical tree structure as shown in Figure 1 (right). Gong et al (2017) propose several MWS approaches and show that treating MWS as constituent parsing leads to the best performance. They adopt the transition-based parser of Cross and Huang (2016), which greedily searches an optimal shift-reduce action sequence to build a tree.…”

“…They adopt the transition-based parser of Cross and Huang (2016), which greedily searches an optimal shift-reduce action sequence to build a tree. In this work, instead of adopting the transition-based parser as Gong et al (2017), we employ the graph-based parser of Stern et al (2017) and replace the original global max-margin loss with local span-wise loss (Joshi et al, 2018;Teng and Zhang, 2018) as our basic MWS model due to two considerations: 1) the graph-based parser with local loss gains more efficiency without hurting the performance compared with the transitionbased parser and the graph-based parser with global loss, which will be discussed in Section 5.3; 2) Figure 2: Architecture of our MWS model. more importantly, this work aims to conduct in-depth study on a simple, efficient, and effective way to incorporate weakly labeled data for MWS.…”

“…Due to the lack of manually labeled high-quality data, Gong et al (2017) construct a large-scale pseudo labeled MWS data by automatically converting existing heterogeneous SWS data in a pairwise way. They train their model with the pseudo labeled data and obtain promising performance on a small scale manually labeled MWS data.…”

“…Motivated by above perspectives, multi-grained word segmentation (MWS) is formally proposed by Gong et al (2017) as a useful and challenging direction for research on word segmentation. Given an input sentence, MWS aims to accommodate all words of different granularities with a hierarchical tree structure.…”

RethinkCWS: Is Chinese Word Segmentation a Solved Task?

“…The changing of F1-score as more source domains are introduced in three different orders: Max-, Min-, and Randselect. The red dotted line is the result reported byChen et al (2017) with the same model, trained on nine datasets. 1…”