Unsupervised Neural Dependency Parsing

Abstract

Unsupervised dependency parsing aims to learn a dependency grammar from text annotated with only POS tags. Various features and inductive biases are often used to incorporate prior knowledge into learning. One useful type of prior information is that there exist correlations between the parameters of grammar rules involving different POS tags. Previous work employed manually designed features or special prior distributions to encode such information. In this paper, we propose a novel approach to unsupervised dependency parsing that uses a neural model to predict grammar rule probabilities based on distributed representation of POS tags. The distributed representation is automatically learned from data and captures the correlations between POS tags. Our experiments show that our approach outperforms previous approaches utilizing POS correlations and is competitive with recent state-of-the-art approaches on nine different languages.

Dependency Grammar Induction with Neural Lexicalization and Big Training Data

“…WSJ10 WSJ Unlexicalized Approaches, with WSJ10 EVG (Headden III et al, 2009) 65.0 -TSG-DMV (Blunsom and Cohn, 2010) 65.9 53.1 PR-S (Gillenwater et al, 2010) 64.3 53.3 HDP-DEP (Naseem et al, 2010) 73.8 -UR-A E-DMV (Tu and Honavar, 2012) 71.4 57.0 Neural E-DMV (Jiang et al, 2016) 72.5 57.6 Systems Using Lexical Information and/or More Data LexTSG-DMV (Blunsom and Cohn, 2010) 67.7 55.7 L-EVG (Headden III et al, 2009) 68.8 -CS (Spitkovsky et al, 2013) 72.0 64.4 MaxEnc (Le and Zuidema, 2015) 73. Again, we find that good initialization leads to better performance than KM initialization, and both good initialization and KM initialization are significantly better than random and uniform initialization.…”

“…Here we employ a neural approach to smoothing. Specifically, we propose a lexicalized extension of neural DMV (Jiang et al, 2016) and we call the resulting approach L-NDMV.…”

“…It was previously shown that the heuristic KM initialization method by Klein and Manning (2004) does not work well for lexicalized grammar induction (Headden III et al, 2009;Pate and Johnson, 2016) and it is very helpful to initialize learning with a model learned by a different grammar induction method (Le and Zuidema, 2015;Jiang et al, 2016). We tested both KM initialization and the following initialization method: we first learn an unlexicalized DMV using the grammar induction method of Naseem et al (2010) and use it to parse the training corpus; then, from the parse trees we run maximum likelihood estimation to produce the initial lexicalized model.…”

“…Finally, smoothing techniques can be used to reduce the negative impact of data scarcity. One example is Neural DMV (NDMV) (Jiang et al, 2016) which incorporates neural networks into DMV and can automatically smooth correlated grammar rule probabilities.…”

CRF Autoencoder for Unsupervised Dependency Parsing

“…where D(x) is the set of all possible dependency arcs of sentence x, 1[·] is the indicator function, and µ(x, i, j) is the expected count defined as follows, (Jiang et al, 2016), and Convex-MST (Grave and Elhadad, 2015) Methods WSJ10 WSJ Basic Setup Feature DMV (Berg-Kirkpatrick et al, 2010) 63.0 -UR-A E-DMV (Tu and Honavar, 2012) 71.4 57.0 Neural E-DMV (Jiang et al, 2016) 69.7 52.5 Neural E-DMV (Good Init) (Jiang et al, 2016) 72.5 57.6 Basic Setup + Universal Linguistic Prior Convex-MST (Grave and Elhadad, 2015) 60.8 48.6 HDP-DEP (Naseem et al, 2010) 71.9 -CRFAE 71.7 55.7 Systems Using Extra Info LexTSG-DMV (Blunsom and Cohn, 2010) 67.7 55.7 CS (Spitkovsky et al, 2013) 72.0 64.4 MaxEnc (Le and Zuidema, 2015) 73.2 65.8 Table 3: Comparison of recent unsupervised dependency parsing systems on English. Basic setup is the same as our setup except that linguistic prior is not used.…”