On the Role of Supervision in Unsupervised Constituency Parsing

Abstract

We analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing F 1 score on the Wall Street Journal (WSJ) development set (1,700 sentences). We introduce strong baselines for them, by training an existing supervised parsing model (Kitaev and Klein, 2018) on the same labeled examples they access. When training on the 1,700 examples, or even when using only 50 examples for training and 5 for development, such a few-shot parsing approach can outperform all the unsupervised parsing methods by a significant margin. Fewshot parsing can be further improved by a simple data augmentation method and selftraining. This suggests that, in order to arrive at fair conclusions, we should carefully consider the amount of labeled data used for model development. We propose two protocols for future work on unsupervised parsing: (i) use fully unsupervised criteria for hyperparameter tuning and model selection; (ii) use as few labeled examples as possible for model development, and compare to few-shot parsing trained on the same labeled examples. 1

Substructure Substitution: Structured Data Augmentation for NLP

“…BiAffine: the bi-affine dependency parsing model proposed by Dozat and Manning (2017). Shi et al (2020) have shown that SUB 2 can significantly improve few-shot constituency parsing on the Penn Treebank dataset; in this work, we extend the few-shot parsing evaluation to other domains, using the Foreebank (FBANK; Kaljahi et al, 2015) and NXT-Switchboard (SWBD;Calhoun et al, 2010) datasets. Foreebank consists of 1,000 English and 1,000 French sentences; for either language, we randomly select 50 sentences for training, 50 for development, and 250 for testing.…”

“…Data augmentation has been found effective for various natural language processing (NLP) tasks, such as machine translation (Fadaee et al, 2017;Gao et al, 2019;Xia et al, 2019, inter alia), text classification (Wei and Zou, 2019;Quteineh et al, 2020), syntactic and semantic parsing (Jia and Liang, 2016;Shi et al, 2020;Dehouck and Gómez-Rodríguez, 2020), semantic role labeling (Fürstenau and Lapata, 2009), and dialogue understanding (Hou et al, 2018;Niu and Bansal, 2019). Such methods enhance the diversity of the training set by generating examples based on existing ones, and can make the learned models more robust against noise (Xie et al, 2020).…”

“…SUB 2 also falls into this category. The idea of same-label substructure substitution has been used to improve performance on structured prediction tasks such as semantic parsing (Jia and Liang, 2016), constituency parsing (Shi et al, 2020), dependency parsing (Dehouck and Gómez-Rodríguez, 2020), named entity recognition (Dai and Adel, 2020), meaning representation-based text generation (Kedzie and McKeown, 2020), and compositional generalization (Andreas, 2020). To the best of our knowledge, however, SUB 2 has not been systematically studied as a general data augmentation method for NLP tasks.…”

Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models

“…We here mention a few limitations of our approach and propose avenues for future work. First, analogous to several unsupervised parsers (Shi et al, 2020) including PCFGs (Zhao and Titov, 2021), the current form of our method relies on a few gold-standard annotations from the validation set to determine the best hyperparameters (i.e., the best choice for attention head selection). This dependency makes it hard to say that our approach is entirely unsupervised, although it steps aside from a typical way of learning parsers with supervision.…”

“…Notably, developed an approach similar to ours, but they focused on Transformers trained for machine translation rather than language models. Work on neural unsupervised parsing (Shen et al (2018b(Shen et al ( , 2019; Kim et al (2019a); Shi et al (2020), inter alia) also seeks to generate parse trees without supervision from gold-standard trees. It is worth noting that some work such as Kann et al (2019) and Zhao and Titov (2021) attempt to evaluate unsupervised parsers in multilingual settings, akin to our work.…”