Noisy-context surprisal as a human sentence processing cost model

Futrell, Richard; Lévy, Roger

doi:10.18653/v1/e17-1065

Cited by 54 publications

(64 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are good theoretical and empirical reasons to believe the HDMI hypothesis is true. The empirically-measured mutual information of words pairs in head-dependent relationships has been found to be greater than various baselines in Futrell and Levy (2017) across languages. Theoretically, it makes sense for word pairs in dependency relationships to have the highest mutual information because mutual information is a measure of the strength of covariance between two variables, and words in dependencies are by definition those word pairs whose covariance is directly constrained by grammatical rules.…”

Section: Dlm As An Approximation To Information Localitymentioning

confidence: 76%

See 1 more Smart Citation

Information-theoretic locality properties of natural language

Futrell¹

2019

Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019)

Self Cite

View full text Add to dashboard Cite

I present theoretical arguments and new empirical evidence for an information-theoretic principle of word order: information locality, the idea that words that strongly predict each other should be close to each other in linear order. I show that information locality can be derived under the assumption that natural language is a code that enables efficient communication while minimizing information-processing costs involved in online language comprehension, using recent psycholinguistic theories to characterize those processing costs information-theoretically. I argue that information locality subsumes and extends the previously-proposed principle of dependency length minimization (DLM), which has shown great explanatory power for predicting word order in many languages. Finally, I show corpus evidence that information locality has improved explanatory power over DLM in two domains: in predicting which dependencies will have shorter and longer lengths across 50 languages, and in predicting the preferred order of adjectives in English.

show abstract

Section: Dlm As An Approximation To Information Localitymentioning

confidence: 76%

“…3 Lossy-context surprisal and information locality I propose to modify surprisal theory in the manner described in Futrell and Levy (2017). The contents of this section are a simplified exposition of the derivations presented in that paper.…”

Section: Background: Efficient Communication Under Information Procesmentioning

confidence: 99%

Information-theoretic locality properties of natural language

Futrell¹

2019

Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The interaction between past exposure and working memory is an active area of research. Recent extensions to the surprisal metric (Hale, 2001; Levy, 2008) such as the lossy‐context surprisal (Futrell et al, 2019; Futrell & Levy, 2017) have been shown to predict locality as well as the forgetting effects pattern in English and German. Certain models of working memory that can perform differential resource allocation based on prior experience (Daily, Lovett, & Reder, 2001; Lovett, Daily, & Reder, 2000) also hold promise in accounting for the interaction of limited working memory capacity and past exposure.…”

Section: Discussionmentioning

confidence: 99%

“…Both these factors have been shown to be critical in determining online processing cost (Staub, 2010). And while there are some proposals that have tried to investigate the interaction between the two factors (Campanelli et al, 2018; Demberg, Keller, & Koller, 2013; Futrell & Levy, 2017; Husain & Vasishth, 2014; Husain et al, 2014; Levy & Keller, 2013; Vasishth & Drenhaus, 2011), a formal account of such an interaction over naturalistic data is currently lacking. As we will note later, our work, in fact, points to such an interaction between linguistic exposure and dependency minimization.…”

Section: Introductionmentioning

confidence: 99%

Word Order Typology Interacts With Linguistic Complexity: A Cross‐Linguistic Corpus Study

et al. 2020

View full text Add to dashboard Cite

Much previous work has suggested that word order preferences across languages can be explained by the dependency distance minimization constraint (Ferrer-i Cancho, 2008Hawkins, 1994). Consistent with this claim, corpus studies have shown that the average distance between a head (e.g., verb) and its dependent (e.g., noun) tends to be short cross-linguistically show that the comprehension system can adapt to the typological properties of a language, for example, verb-final order, leading to more complex structures, for example, having longer linear distance between a head and its dependent. In this paper, we conduct a corpus study for a group of 38 languages, which were either Subject-Verb-Object (SVO) or Subject-Object-Verb (SOV), in order to investigate the role of word order typology in determining syntactic complexity. We present results aggregated across all dependency types, as well as for specific verbal (objects, indirect objects, and adjuncts) and nonverbal (nominal, adjectival, and adverbial) dependencies. The results suggest that dependency distance in a language is determined by the default word order of a language, and crucially, the direction of a dependency (whether the head precedes the dependent or follows it; e.g., whether the noun precedes the verb or follows it). Particularly we show that in SOV languages (e.g., Hindi, Korean) as well as SVO languages (e.g., English, Spanish), longer linear distance (measured as number of words) between head and dependent arises in structures when they mirror the default word order of the language. In addition to showing results on linear distance, we also investigate the influence of word order typology on hierarchical distance (HD; measured as number of heads between head and dependent). The results for HD are similar to that of linear distance. At the same time, in comparison to linear distance, the influence of adaptability on HD seems less strong. In particular, the results Correspondence should be sent to Himanshu Yadav is now affiliated with University of Potsdam, Germany. Vishakha Shukla is now affiliated with Shroff Charity Eye Hospital, Delhi, India. show that most languages tend to avoid greater structural depth. Together, these results show evidence for "limited adaptability" to the default word order preferences in a language. Our results support a large body of work in the processing literature that highlights the importance of linguistic exposure and its interaction with working memory constraints in determining sentence complexity. Our results also point to the possible role of other factors such as the morphological richness of a language and a multifactor account of sentence complexity remains a promising area for future investigation.

show abstract

“…Most recently, Hahn, Degen, Goodman, Jurafsky, and Futrell (2018) show that mutual information contributes to explaining adjective ordering preferences gathered from English speakers; when multiple adjectives are present in a phrase, the adjective with higher mutual information tends to be closer to the noun. Following Futrell and Levy (2017), this could be driven by memory-constrained incremental processing. Placing modifier-noun pairs with high mutual information far apart from one another increases processing effort.…”

Section: Discussionmentioning

confidence: 99%

From the world to word order: the link between conceptual structure and language

Culbertson¹,

Schouwstra²,

Kirby³

2019

Preprint

View full text Add to dashboard Cite

The world’s languages exhibit striking diversity. At the same time, recurring linguistic patterns suggest the possibility that this diversity is shaped by features of human cognition. One well-studied example is word order in complex noun phrases (like 'these two red vases'). While many orders of these elements are possible, a subset appear to be preferred. It has been argued that this order reflects a single underlying representation of noun phrase structure, from which preferred orders are straightforwardly derived (e.g., Cinque, 2005). Building on previous experimental evidence using artificial language learning, we show that these preferred orders arise not only in existing languages, but also in improvised sequences of gestures produced by English speakers. We then use corpus data from a wide range of languages to argue that the hypothesized underlying structure of the noun phrase might be learnable from statistical features relating objects and their properties conceptually. Using an information-theoretic measure of strength of association, we find that adjectival properties (e.g. 'red') are on average more closely related to the objects they modify (e.g. 'wine'), than numerosities (e.g. 'two'), which are in turn more closely related to the objects they modify than demonstratives (e.g., 'this'). It is exactly those orders which transparently reflect this—by placing adjectives closest to the noun, and demonstratives farthest away—which are more common across languages, and preferred in our silent gesture experiments. These results suggest that our experience with objects in the world, combined with a preference for transparent mappings from conceptual structure to linear order, can explain constraints on noun phrase order.

show abstract

Noisy-context surprisal as a human sentence processing cost model

Cited by 54 publications

References 31 publications

Information-theoretic locality properties of natural language

Information-theoretic locality properties of natural language

Word Order Typology Interacts With Linguistic Complexity: A Cross‐Linguistic Corpus Study

From the world to word order: the link between conceptual structure and language

Contact Info

Product

Resources

About