Co-Learning for Few-Shot Learning

Xu, Rui; Liu, Xing; Shao, Shuai; Liu, Baodi; Zhang, Kai; Liu, Weifeng

doi:10.1007/s11063-022-10770-4

Cited by 5 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…performance for 5M , 10M , 30M model sizes without additional training, and (iii) improves or performs similarly to AutoDistil[22] for 5M , 26.8M , and 50M model sizes. Compared to HAT[19], our best supernet: (i) reduces the supernet vs. the standalone model gap by 26.5%, (ii) yields a better pareto front for latency-BLEU tradeoff (100 to 200 ms), and (iii) reduces the number of additional steps to close the gap by 39.8%.…”

mentioning

confidence: 82%

“…BERT supernet and standalone are pretrained from scratch on Wikipedia and Books Corpus [30]. We evaluate the performance of the BERT model by finetuning on each of the seven tasks (chosen by AutoDistil [22]) in the GLUE benchmark [18]. The data preprocessing, pretraining settings, and finetuning settings are discussed in A.1.1.…”

Section: Experiments Setupmentioning

confidence: 99%

“…The state-of-the-art NAS frameworks for building a task-agnostic BERT model are NAS-BERT [23] and AutoDistil [22]. 4 The NAS-BERT pipeline includes: (1) supernet training (with a Transformer stack containing multi-head attention, feed-forward network [FFN] and convolutional layers in arbitrary positions), (2) search based on the distillation (task-agnostic) loss, and (3) pretraining the best architecture from scratch (1x pretraining budget, which is 2048 batch size * 125,000 steps).…”

Section: Comparison With Sota Nasmentioning

confidence: 99%

“…The authors find that SPOS performs poorly compared to the sandwich training objective. AutoDistil [22] employs few-shot NAS [28]: construct K search spaces of non-overlapping BERT architectures and train a weight-sharing BERT supernet for each search space. The search is based on self-attention distillation loss with BERT-Base (task-agnostic search) and MNLI score (proxy search).…”

Section: Related Workmentioning

confidence: 99%

“…Neural architecture search (NAS) can automatically design architectures that achieve high quality on the natural language processing (NLP) task, while satisfying user-defined efficiency (e.g., latency, memory) constraints [19,23,22]. Most straightforward way of NAS is treating it as the black-box optimization [31,14].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Automatic Detection of Machine Generated Text: A Critical Survey

Jawahar¹,

Abdul-Mageed²,

Lakshmanan³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Text generative models (TGMs) excel in producing text that matches the style of human language reasonably well. Such TGMs can be misused by adversaries, e.g., by automatically generating fake news and fake product reviews that can look authentic and fool humans. Detectors that can distinguish text generated by TGM from human written text play a vital role in mitigating such misuse of TGMs. Recently, there has been a flurry of works from both natural language processing (NLP) and machine learning (ML) communities to build accurate detectors for English. Despite the importance of this problem, there is currently no work that surveys this fast-growing literature and introduces newcomers to important research challenges. In this work, we fill this void by providing a critical survey and review of this literature to facilitate a comprehensive understanding of this problem. We conduct an in-depth error analysis of the state-of-the-art detector and discuss research directions to guide future work in this exciting area.

show abstract

mentioning

confidence: 82%