Neural Network for Heterogeneous Annotations

Abstract

Multiple treebanks annotated under heterogeneous standards give rise to the research question of best utilizing multiple resources for improving statistical models. Prior research has focused on discrete models, leveraging stacking and multi-view learning to address the problem. In this paper, we empirically investigate heterogeneous annotations using neural network models, building a neural network counterpart to discrete stacking and multiview learning, respectively, finding that neural models have their unique advantages thanks to the freedom from manual feature engineering. Neural model achieves not only better accuracy improvements, but also an order of magnitude faster speed compared to its discrete baseline, adding little time cost compared to a neural model trained on a single treebank.

Universal Dependencies Parsing for Colloquial Singaporean English

“…In order to obtain automatically predicted POS tags as features for a base English dependency parser, we train a POS tagger for UD-Eng using the baseline model of Chen et al (2016), depicted in Figure 3. The bi-LSTM networks with a CRF layer (bi-LSTM-CRF) have shown state-of-the-art performance by globally optimizing the tag sequence Chen et al, 2016). Based on this English POS tagging model, we train a POS tagger for Singlish using the featurelevel neural stacking model of Chen et al (2016).…”

“…The bi-LSTM networks with a CRF layer (bi-LSTM-CRF) have shown state-of-the-art performance by globally optimizing the tag sequence Chen et al, 2016). Based on this English POS tagging model, we train a POS tagger for Singlish using the featurelevel neural stacking model of Chen et al (2016). Both the English and Singlish models consist of an input layer, a feature layer, and an output layer.…”

“…This overall approach is illustrated in Figure 1. In particular, we train a basic Singlish parser with the best off-the-shelf neural dependency parsing model using biaffine attention (Dozat and Manning, 2017), and improve it with knowledge transfer by adopting neural stacking (Chen et al, 2016;Zhang and Weiss, 2016) to integrate the English syntax. Since POS tags are important features for dependency parsing (Chen and Manning, 2014;, we train a POS tagger for Singlish following the same idea by integrating English POS knowledge using neural stacking.…”

“…Neural stacking was previously used for cross-annotation (Chen et al, 2016) and crosstask (Zhang and Weiss, 2016) joint-modelling on monolingual treebanks. To the best of our knowledge, we are the first to employ it on cross-lingual feature transfer from resource-rich languages to improve dependency parsing for low-resource languages.…”

Switch-LSTMs for Multi-Criteria Chinese Word Segmentation

“…Zoph and Knight;Johnson et al (2016; have been jointly training translation models from and to different languages, it is achieved simply by jointly train encoder or both encoder and decoder. (Jiang, Huang, and Liu 2009;Sun and Wan 2012;Qiu, Zhao, and Huang 2013;Li et al 2015;Chen, Zhang, and Liu 2016) adopted the stack-based model to take advantage of annotated data from multiple sources, and show that tasks can indeed help improve each other.…”