Will it Unblend?

Pinter, Yuval; Jacobs, Cassandra L.; Eisenstein, Jacob

doi:10.18653/v1/2020.findings-emnlp.138

Cited by 7 publications

(7 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The topic of this paper is related to the more fundamental question of how PLMs represent the meaning of complex words in the first place. So far, most studies have focused on methods of representation extraction, using ad-hoc heuristics such as averaging the subword embeddings (Pinter et al, 2020;Sia et al, 2020; or taking the first subword embedding (Devlin et al, 2019;Heinzerling and Strube, 2019;Martin et al, 2020). While not resolving the issue, we lay the theoretical groundwork for more systematic analyses by showing that PLMs can be regarded as serial dual-route models (Caramazza et al, 1988), i.e., the meanings of complex words are either stored or else need to be computed from the subwords.…”

Section: Introductionmentioning

confidence: 99%

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words

Hofmann¹,

Pierrehumbert²,

Schuze³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic probing tasks on which Del-BERT (Derivation leveraging BERT), a model with derivational input segmentation, substantially outperforms BERT with WordPiece segmentation. Our results suggest that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used.

show abstract

Section: Introductionmentioning

confidence: 99%

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words

Hofmann¹,

Pierrehumbert²,

Schuze³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Our experiment shows that popular word form encoders, such as ELMo or BERT's WordPiece, still have a long way to go in terms of recognizing the origins of a novel form. Errors at this stage might lead to inability to handle morphologically complex OOVs in downstream semantic applications (Pinter et al, 2020), although further study of such effects and of the utility of OOV classification in alleviating them is still necessary. Properly leveraging context for morphological decomposition of complex forms also remains an open problem.…”

Section: Discussionmentioning

confidence: 99%

NYTWIT: A Dataset of Novel Words in the New York Times

Pinter

Jacobs

Bittker³

2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We present the New York Times Word Innovation Types dataset, or NYTWIT, a collection of over 2,500 novel English words published in the New York Times between November 2017 and March 2019, manually annotated for their class of novelty (such as lexical derivation, dialectal variation, blending, or compounding). We present baseline results for both uncontextual and contextual prediction of novelty class, showing that there is room for improvement even for state-of-the-art NLP systems. We hope this resource will prove useful for linguists and NLP practitioners by providing a real-world environment of novel word appearance.

show abstract

“…However, to the best of our knowledge, no work to date has explored how Transformer-based encoders represent compounds. The most relevant study in this direction is the one by Pinter et al (2020) reported an overall high similarity between the compound and the constituents, slightly increasing through the layers.…”

Section: Word Representation In Transformersmentioning

confidence: 96%

A Psycholinguistic Analysis of BERT's Representations of Compounds

Buijtelaar¹,

Pezzelle²

2023

Preprint

View full text Add to dashboard Cite

This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e.g., sunlight) whose overall meaning depends-to a various extent-on the semantics of the constituent words (sun, light). We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD; quantifying the weight of each constituent toward the compound meaning) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents' semantics). We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for 'standard' words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide insights into how speakers represent compounds.

show abstract

Will it Unblend?

Cited by 7 publications

References 33 publications

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words

NYTWIT: A Dataset of Novel Words in the New York Times

A Psycholinguistic Analysis of BERT's Representations of Compounds

Contact Info

Product

Resources

About