Linguistic representations in multi-task neural networks for ellipsis resolution

Abstract

Sluicing resolution is the task of identifying the antecedent to a question ellipsis. Antecedents are often sentential constituents, and previous work has therefore relied on syntactic parsing, together with complex linguistic features. A recent model instead used partial parsing as an auxiliary task in sequential neural network architectures to inject syntactic information. We explore the linguistic information being brought to bear by such networks, both by defining subsets of the data exhibiting relevant linguistic characteristics, and by examining the internal representations of the network. Both perspectives provide evidence for substantial linguistic knowledge being deployed by the neural networks.

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

“…Ettinger et al (2016Ettinger et al ( , 2017; Zhu et al (2018), i.a., use a task-based approach similar to ours, where tasks that require a specific subset of linguistic knowledge are used to perform qualitative evaluation. Gulordava et al (2018), Giulianelli et al (2018), Rønning et al (2018), and Jumelet and Hupkes (2018) make a focused contribution towards a particular linguistic phenomenon (agreement, ellipsis, negative polarity). Using recast NLI, Poliak et al (2018a) probe for semantic phenomena in neural machine translation encoders.…”

Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop

“…Wilcox et al (2018) found that two RNN LMs showed significant awareness of such gaps, as well as of some (but not all) of the restrictions on their distribution ("island constraints"; Ross 1967). Rønning et al (2018) analyzed bidirectional RNNs trained to resolve the antecedent of a "sluiced" question word, as in If this is not practical, explain why ; here the phrase this is not practical needs to be interpreted again following the question word, even though it does not reappear. They report both behavioral analyses on challenging cases and analyses of the activations of the network; for example, they show that the distance between the embedding of the question word (why) and the network's activation is minimal at the edge of the antecedent, and conclude that the network reactivates the question word when it encounters the antecedent (recall that the network processes the sentence both from right to left and from left to right).…”

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)

“…Ettinger et al (2016,2017); Zhu et al (2018), i.a., use a task-based approach similar to ours, where tasks that require a specific subset of linguistic knowledge are used to perform qualitative evaluation. Gulordava et al (2018), Giulianelli et al (2018), Rønning et al (2018), and Jumelet and Hupkes ( 2018) make a focused contribution towards a particular linguistic phenomenon (agreement, ellipsis, negative polarity). Using recast NLI, probe for semantic phenomena in neural machine translation encoders.…”