Dynamic Neural Networks: A Survey

Han, Yizeng; Huang, Gao; Song, Sejun; Yang, Le; Wang, Honghui; Wang, Yulin

doi:10.1109/tpami.2021.3117837

Cited by 358 publications

(125 citation statements)

References 198 publications

Supporting

Mentioning

122

Contrasting

Unclassified

Order By: Relevance

“…Since dynamic visual models usually have better efficiency, generality and compatibility, then have attracted researchers' attention in recent years. Dynamic visual models can be roughly categorized as three types [681]: sample-wise, spatial-wise and temporal-wise. Sample-wise dynamic networks [682,683] adjust the architectures (depth, width, etc.)…”

Section: Dynamic Visual Modelsmentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

View full text Add to dashboard Cite

domains indexed by Google News. It contains 31 million documents with an average length of 793 BPE tokens. Like C4, it excludes examples with duplicate URLs. News dumps from December 2016 through March 2019 were used as training data, articles published in April 2019 from the April 2019 dump were used for evaluation. OpenWebText2(OWT2). OWT2 is an enhanced version of the original OpenWebTextCorpus, including content from multiple languages, document metadata, multiple dataset versions, and open source replication code, covering all Reddit submissions from 2005 up until April 2020. PubMed Central(PMC). PMC is a free full-text archive of biomedical and life sciences journal literature from the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The dataset is updated daily. In addition to full-text articles, they contain corrections, retractions, and expressions of concern, as well as file lists that include metadata for articles in each dataset.PMC obtained by open registration in Amazon Web Services (AWS) includes The PMC Open Access Subset and The Author Manuscript Dataset. The PMC Open Access Subset includes all articles and preprints in PMC with a machine-readable Creative Commons license that allows reuse. The Author Manuscript Dataset includes accepted author manuscripts collected under a funder policy in PMC and made available in machine-readable formats for text mining. ArXiv. ArXiv is a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more. It provides open access to academic articles, covering many subdisciplines from vast branches of physics to computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics, which is helpful to the potential downstream applications of the research field. In addition, the writing language of LaTeX also contributes to the study of language models. Colossal Clean Crawled Corpus(C4). C4 is a colossal, cleaned version of Common Crawl's web crawl corpus. It is based on Common Crawl dataset and was used to train the T5 text-to-text Transformer models. The cleaned English version of C4 has 364,868,901 training examples and 364,608 validation examples, while the uncleaned English version has 1,063,805,324 training examples and 1,065,029 validation examples; the realnewslike version has 13,799,838 training examples and 13,863 validation examples, while the webtextlike version has 4,500,788 training examples and 4,493 validation examples. Wiki-40B. Wikipedia (Wiki-40B) is a clean-up text collection containing more than 40 Wikipedia language editions of pages corresponding to entities. The dataset is split into train/validation/test sets for each language. The training set has 2,926,536 examples, the validation set has 163,597 examples, and the test set has 162,274 examples. Wiki-40B is cleaned by a page filter to remove ambiguous, redirected, deleted, and non-physical pages. CLUECorpus2020. CLUECorpus2020 ...

show abstract

Section: Dynamic Visual Modelsmentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In this survey, we describe the overview of general dynamic architectures. If you are interested in other dynamic features, you can find surveys focusing on dynamic networks (Han et al, 2021b). Networks with dynamic architecture can be classified into the following classes:…”

Section: Dynamic Weight Sharingmentioning

confidence: 99%

A Survey on Green Deep Learning

Xu¹,

Zhou²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV). However, despite promising results, it needs to be noted that the computations required by SOTA models have been increased at an exponential rate. Massive computations not only have a surprisingly large carbon footprint but also have negative effects on research inclusiveness and deployment on real-world applications. Green deep learning is an increasingly hot research field that appeals to researchers to pay attention to energy usage and carbon emission during model training and inference. The target is to yield novel results with lightweight and efficient technologies. Many technologies can be used to achieve this goal, like model compression and knowledge distillation. This paper focuses on presenting a systematic review of the development of Green deep learning technologies. We classify these approaches into four categories: (1) compact networks, (2) energy-efficient training strategies, (3) energy-efficient inference approaches, and (4) efficient data usage. For each category, we discuss the progress that has been achieved and the unresolved challenges.

show abstract

“…Dynamic Modeling: Dynamic modeling which the model structures or parameters can be adapted to the input during inference is also another effective approach to reduce the computational cost (Han et al, 2021b). Xin et al (2020a) employed the idea of early exiting as in DeeBERT (Xin et al, 2020b) more computations and the confidence score of positive documents is not only the exiting criterion but the score for re-ranking.…”

Section: Model Accelerationmentioning

confidence: 99%

Pre-training Methods in Information Retrieval

Fan¹,

Xie²,

Cai³

et al. 2021

Preprint

View full text Add to dashboard Cite

The core of information retrieval (IR) is to identify relevant information from large-scale resources and return it as a ranked list to respond to user's information need. Recently, the resurgence of deep learning has greatly advanced this field and leads to a hot topic named NeuIR (i.e., neural information retrieval), especially the paradigm of pre-training methods (PTMs). Owing to sophisticated pre-training objectives and huge model size, pre-trained models can learn universal language representations from massive textual data, which are beneficial to the ranking task of IR. Since there have been a large number of works dedicating to the

show abstract

Dynamic Neural Networks: A Survey

Cited by 358 publications

References 198 publications

A Roadmap for Big Model

A Roadmap for Big Model

A Survey on Green Deep Learning

Pre-training Methods in Information Retrieval

Contact Info

Product

Resources

About