The emergence of number and syntax units in

Abstract

Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as longdistance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that longdistance number information is largely managed by two "number units". Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.

What Limits Our Capacity to Process Nested Long-Range Dependencies in Sentence Comprehension?

“…For example, while in humans, the occurrence of an attractor within a prepositional phrase was found to elicit more agreement errors compared to its presence in a relative clause [ 4 ], recurrent neural network language models were found to show the opposite effect [ 35 ]. Similarly, recurrent neural models were found in Reference [ 33 ] to produce different errors rates than humans on a variety of syntactic structures (but see, References [ 26 , 36 ]).…”

“…In a recent work [ 26 ], we studied the processing of long-range dependencies in an NLM and found that a sparse neural circuit emerged in the neural language model during training, which was shown to carry grammatical-number agreements across long-range dependencies in various sentences. In this section, we briefly describe the main findings therein, and describe how a capacity limitation can further be derived.…”

“…See also Figure 3 in Ref. [ 26 ]). Specifically, the activity of this syntax unit is positive throughout the main subject-verb dependency and changes sign only at the main verb.…”

Neural Language Models Capture Some, But Not All, Agreement Attraction Effects

“…as grammatical. Prior work has shown that LSTMs seem to use an maintain an encoding of the subject's number to process agreement (Lakretz et al, 2019); the fact that this asymmetry emerged from what appears to be an encoding model of agreement is therefore particularly intriguing and should motivate future work.…”

Probing Word and Sentence Embeddings for Long-distance Dependencies Effects in French and English

“…Initial work had shown RNNs do not really learn the structure-dependency of this construction (Linzen et al, 2016), but follow up work has shown that stronger techniques can yield more positive results (Gulordava et al, 2018), only to be very promptly rebutted by work suggesting that the apparently positive results could be the artifact of a much simpler strategy, which takes advantage of the unnaturally simple structure of the examples and simply learns properties of the first word in the sentence (Kuncoro et al, 2018). Recent work by Lakretz et al (2019), however, studies RNNs in more detail, looking at single neurons, and finds that individual neurons encode linguistically meaningful features very saliently and with behaviour over time that corresponds to the expected propagation of subject-verb number agreement information.…”