Active Curriculum Learning

Jafarpour, Borna; Sepehr, Dawn; Pogrebnyakov, Nick

doi:10.18653/v1/2021.internlp-1.6

Cited by 7 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Curriculum design greatly varies in each work. Linguistic features that have been used in curriculum formation include Parts-of-Speech (POS) information, n-gram frequency (Platanios et al, 2019), average number of dependents per word in the sentence parse tree (Jafarpour et al, 2021), edit distance (Kadotani et al, 2021;Chang et al, 2021). However, arguably, the most common curriculum formations are based on measures of frequency (Liu et al, 2018) and text length (Tay et al, 2019;Cirik et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Mmi01 at The BabyLM Challenge: Linguistically Motivated Curriculum Learning for Pretraining in Low-Resource Settings

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

This paper presents our findings for the BabyLM Challenge (Warstadt et al., 2023). Our exploration is inspired by vanilla curriculum learning (Bengio et al., 2009) and we explored the effect of linguistic complexity in forming the best curriculum for pre-training. In particular, we explore curriculum formations based on dependency-based measures (dependents per token, average dependency distance) and lexical-based measures (rarity, density, dispersion and diversity). We found that, overall, models pretrained using curriculum learning were able to beat the performance of a noncurriculum learning pre-trained model. Furthermore, we notice using different linguistic metric for measuring complexity lead to advantageous performance for some tasks, but not all. We share our results and analysis in the hope that it can provide beneficial insights for future work.

show abstract

Section: Related Workmentioning

confidence: 99%

Mmi01 at The BabyLM Challenge: Linguistically Motivated Curriculum Learning for Pretraining in Low-Resource Settings

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

show abstract

“…Curriculum learning (CL) is widely used in reinforcement learning (Zaremba and Sutskever, 2014;Matiisen et al, 2019;Graves et al, 2017) and neural machine translation (Platanios et al, 2019;Kocmi and Bojar, 2017;Guo et al, 2020). Sachan and Xing (2018), Penha and Hauff (2020), and Jafarpour et al (2021) demonstrate the effectiveness of CL for question generation, information retrieval, natural language understanding and named entity recognition respectively. To the best of our knowledge, we are the first to examine the efficacy of CL for commonsense reasoning.…”

Section: Related Workmentioning

confidence: 99%

On Curriculum Learning for Commonsense Reasoning

Maharana¹,

Bansal²

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Commonsense reasoning tasks follow a standard paradigm of finetuning pretrained language models on the target task data, where samples are introduced to the model in a random order during training. However, recent research suggests that data order can have a significant impact on the performance of finetuned models for natural language understanding. Hence, we examine the effect of a humanlike easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks. We use paced curriculum learning to rank data and sample training mini-batches with increasing levels of difficulty from the ranked dataset during finetuning. Further, we investigate the effect of an adaptive curriculum, i.e., the data ranking is dynamically updated during training based on the current state of the learner model. We use a teacher model to measure difficulty of each sample and experiment with three measures based on question answering probability, variability and out-ofdistribution. To understand the effectiveness of curriculum learning in various scenarios, we apply it on full model fine-tuning as well as parameter-efficient prompt-tuning settings. Our results show that fixed as well as adaptive curriculum learning significantly improve performance for five commonsense reasoning tasks, i.e., SocialIQA, CosmosQA, CODAH, HellaSwag, WinoGrande in both tuning settings. Further, we find that prioritizing the difficult samples in the tail end of training improves generalization to unseen in-domain data as well as out-of-domain data. Our work provides evidence and encourages research into curriculum learning for commonsense reasoning. 1 .

show abstract

“…We describe our submission to the BabyLM Challenge (Warstadt et al, 2023), a shared-task about language models trained from scratch on a developmentally plausible corpus. Inspired by expectationbased theories of sentence processing (Hale, 2001;Levy, 2008) and active curriculum learning (ACL) (Jafarpour et al, 2021), our approach relies on surprisal to select informative samples and streamline them into the model during training. We henceforth refer to our strategy as active curriculum learning modeling (ACLM).…”

Section: Introductionmentioning

confidence: 99%

“…The uncertainty metrics, however, tend to bias the model towards eccentric examples (Zhang et al, 2022b). To counteract this, Jafarpour et al (2021) use CL, a technique that mimics how humans learn by regulating the training according to some schedule criterion, e.g., easy to difficult or short to long examples (Bengio et al, 2009).…”

Section: Introductionmentioning

confidence: 99%

A surprisal oracle for active curriculum language modeling

Hong,

Loáiciga,

Sayeed

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

We investigate the viability of surprisal in an active curriculum learning framework to train transformer-based language models in the context of the BabyLM Challenge. In our approach, the model itself selects the data to label (active learning) and schedules data samples based on a surprisal oracle (curriculum learning). We show that the models learn across all the tasks and datasets evaluated, making the technique a promising alternative approach to reducing the data requirements of language models. Our code is available at https: //github.com/asayeed/ActiveBaby.

show abstract

Active Curriculum Learning

Cited by 7 publications

References 20 publications

Mmi01 at The BabyLM Challenge: Linguistically Motivated Curriculum Learning for Pretraining in Low-Resource Settings

Mmi01 at The BabyLM Challenge: Linguistically Motivated Curriculum Learning for Pretraining in Low-Resource Settings

On Curriculum Learning for Commonsense Reasoning

A surprisal oracle for active curriculum language modeling

Contact Info

Product

Resources

About