2022
DOI: 10.1162/tacl_a_00500
|View full text |Cite
|
Sign up to set email alerts
|

Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks

Abstract: Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning res… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
1

Citation Types

0
1
0

Year Published

2023
2023
2024
2024

Publication Types

Select...
2
1

Relationship

0
3

Authors

Journals

citations
Cited by 3 publications
(1 citation statement)
references
References 70 publications
(80 reference statements)
0
1
0
Order By: Relevance
“…While we do not test whether models can memorize long-tail knowledge, we instead test whether models can process long-tail sentences. Naik et al (2022) note that it is challenging to catalogue and evaluate generalization along micro-level dimensions and instead propose benchmarks that vary along macro-level dimensions (such as the language and domain) as a proxy. We hypothesize that LMs learn which micro-level phenomena are rare, as this would improve their overall language modeling objective.…”
Section: Related Workmentioning
confidence: 99%
“…While we do not test whether models can memorize long-tail knowledge, we instead test whether models can process long-tail sentences. Naik et al (2022) note that it is challenging to catalogue and evaluate generalization along micro-level dimensions and instead propose benchmarks that vary along macro-level dimensions (such as the language and domain) as a proxy. We hypothesize that LMs learn which micro-level phenomena are rare, as this would improve their overall language modeling objective.…”
Section: Related Workmentioning
confidence: 99%