Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Chu, Xiangxiang; Zhou, Tianbao; Zhang, Bo; Li, Jixiang

doi:10.1007/978-3-030-58555-6_28

Cited by 233 publications

(208 citation statements)

References 15 publications

Supporting

Mentioning

205

Contrasting

Order By: Relevance

“…3.3 1.3 *DropNAS (Hong et al, 2020) 2.58±0.14 4.1 0.6 *PC-DARTS (Xu et al, 2020) 2.57±0.07 3.6 0.1 *FairDARTS (Chu et al, 2019) 2.54 3.3 0.4 *DrNAS 2.54±0.03 4.0 0.4 *GTN 2.92±0.06 8.2 0.67 *GTN(F=128) 2.42±0.03 97.9 0.67 *DARTS-1st 3.00±0.14 3.3 0.4 LBT(RN18,DARTS-1st) (ours) 2.87±0.05 3.2 0.6 LBT(RN50,DARTS-1st) (ours) 2.79±0.07 3.3 0.7 *DARTS-2nd 2.76±0.09 3.3 1.5 LBT(RN18,DARTS-2nd) (ours) 2.65±0.03 3.4 1.9 LBT(RN50,DARTS-2nd) (ours) 2.61±0.05 3.4 2.1 *P-DARTS 2.50 3.4 0.3 LBT(RN18,P-DARTS) (ours) 2.64±0.11 3.4 0.4 LBT(RN50,P-DARTS) (ours) 2.57±0.15 3.4 0.5 *PC-DARTS (Xu et al, 2020) 2.57±0.07 3.6 0.1 LBT(RN18,PC-DARTS) (ours) 2.59±0.03 3.7 0.1 LBT(RN50,PC-DARTS) (ours) 2.56±0.04 3.7 0.2 in the third column and fourth column, the model size and search cost of our methods are similar to those of baselines.…”

Section: Resultsmentioning

confidence: 99%

“…*Inception-v1 (Szegedy et al, 2015) 30.2 10.1 6.6 -*MobileNet (Howard et al, 2017) 29.4 10.5 4.2 -*ShuffleNet 2× (v2) (Ma et al, 2018) 25.1 7.6 7.4 -*NASNet-A 26.0 8.4 5.3 1800 *PNAS (Liu et al, 2018a) 25.8 8.1 5.1 225 *MnasNet-92 (Tan et al, 2019) 25.2 8.0 4.4 1667 *AmoebaNet-C (Real et al, 2019) 24.3 7.6 6.4 3150 *SNAS-CIFAR10 27.3 9.2 4.3 1.5 *BayesNAS-CIFAR10 26.5 8.9 3.9 0.2 *PARSEC-CIFAR10 (Casale et al, 2019) 26.0 8.4 5.6 1.0 *GDAS-CIFAR10 (Dong and Yang, 2019) 26.0 8.5 5.3 0.2 *DSNAS-ImageNet 25.7 8.1 --*SDARTS-ADV-CIFAR10 25.2 7.8 5.4 1.3 *PC-DARTS-CIFAR10 (Xu et al, 2020) 25.1 7.8 5.3 0.1 *ProxylessNAS-ImageNet (Cai et al, 2019) 24.9 7.5 7.1 8.3 *FairDARTS-CIFAR10 (Chu et al, 2019) 24.9 7.5 4.8 0.4 *FairDARTS-ImageNet (Chu et al, 2019) 24.4 7.4 4.3 3.0 *DrNAS-ImageNet 24.2 7.3 5.2 3.9 *DARTS − -ImageNet (Chu et al, 2020a) 23.8 7.0 4.9 4.5 *DARTS + -CIFAR100 23.7 7.2 5.1 0.2 *DARTS-2nd(CIFAR10) 26.7 8.7 4.7 1.5 LBT(RN50,DARTS-2nd,CIFAR10) (ours) 25.5 7.9 4.8 2.1 *P-DARTS(CIFAR10) 24.4 7.4 4.9 0.3 LBT(RN50,P-DARTS,CIFAR10) (ours) 24.1 7.2 4.9 0.5 *P-DARTS(CIFAR100) 24.7 7.5 5.1 0.3 LBT(RN50,P-DARTS,CIFAR100) (ours)…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Learning by Teaching, with Application to Neural Architecture Search

Sheth¹,

Jiang²,

Xie³

2021

Preprint

View full text Add to dashboard Cite

Learning by teaching is a broadly used methodology in human learning and shows great effectiveness in improving learning outcome: a learner deepens his/her understanding of a topic by teaching this topic to others. We are interested in investigating whether this powerful learning technique can be borrowed from humans to improve the learning abilities of machines. We propose a novel machine learning approach called learning by teaching (LBT). In our approach, the teacher creates a pseudo-labeled dataset and uses it to train a student model. Based on how the student performs on the validation dataset, the teacher re-learns its model and re-teaches the student until the student achieves great validation performance. We propose a multi-level optimization framework to formulate LBT which involves three learning stages: teacher learns; teacher teaches student; teacher and student validate themselves. We develop an efficient algorithm to solve the LBT problem. We apply our approach to neural architecture search on CIFAR-100, CIFAR-10, and ImageNet, where the results demonstrate the effectiveness of our method.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Learning by Teaching, with Application to Neural Architecture Search

Sheth¹,

Jiang²,

Xie³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…3.3 1.3 *DropNAS (Hong et al, 2020) 2.58±0.14 4.1 0.6 *PC-DARTS (Xu et al, 2020) 2.57±0.07 3.6 0.1 *FairDARTS (Chu et al, 2019) 2.54 3.3 0.4 *DrNAS 2.54±0.03 4.0 0.4 *P-DARTS 2.50 3.4 0.3 *DARTS-1st 3.00±0.14 3.3 0.4 LPT-R18-DARTS-1st (ours) 2.85±0.09 2.7 0.6 *DARTS-2nd 2.76±0.09 3. (Chu et al, 2020b), and DrNAS .…”

Section: Resultsmentioning

confidence: 99%

“…This shows that LPT is able to search betterperforming architectures without significantly increasing network size and search cost. A few additional remarks: 1) On CIFAR-100, DARTS-2nd with second-order approximation in the optimization algorithm is not advantageous compared with DARTS-1st which uses (Szegedy et al, 2015) 30.2 10.1 6.6 -*MobileNet (Howard et al, 2017) 29.4 10.5 4.2 -*ShuffleNet 2× (v1) 26.4 10.2 5.4 -*ShuffleNet 2× (v2) (Ma et al, 2018) 25.1 7.6 7.4 -*NASNet-A 26.0 8.4 5.3 1800 *PNAS (Liu et al, 2018a) 25.8 8.1 5.1 225 *MnasNet-92 (Tan et al, 2019) 25.2 8.0 4.4 1667 *AmoebaNet-C (Real et al, 2019) 24.3 7.6 6.4 3150 *SNAS 27.3 9.2 4.3 1.5 *BayesNAS 26.5 8.9 3.9 0.2 *PARSEC (Casale et al, 2019) 26.0 8.4 5.6 1.0 *GDAS (Dong and Yang, 2019) 26.0 8.5 5.3 0.2 *DSNAS 25.7 8.1 --*SDARTS-ADV 25.2 7.8 5.4 1.3 *PC-DARTS (Xu et al, 2020) 25.1 7.8 5.3 0.1 *ProxylessNAS (Cai et al, 2019) 24.9 7.5 7.1 8.3 *FairDARTS (Chu et al, 2019) 24.9 7.5 4.8 0.4 *P-DARTS (CIFAR-100) 24.7 7.5 5.1 0.3 *P-DARTS (CIFAR-10) 24.4 7.4 4.9 0.3 *FairDARTS (Chu et al, 2019) 24.4 7.4 4.3 3.0 *DrNAS 24.2 7.3 5.2 3.9 *PC-DARTS (Xu et al, 2020) 24.2 7.3 5.3 3.8 *DARTS + (Liang et al, 2019a) 23.9 7.4 5.1 6.8 *DARTS − (Chu et al, 2020a) 23.8 7.0 4.9 4.5 *DARTS + (CIFAR-100) (Liang et al, 2019a) 23. (Chu et al, 2020a) and DrNAS .…”

Section: Resultsmentioning

confidence: 99%

Skillearn: Machine Learning Inspired by Humans' Learning Skills

Xie¹,

Du²,

Ban³

2020

Preprint

View full text Add to dashboard Cite

<div>Humans, as the most powerful learners on the planet, have accumulated a lot of learning skills, such as learning through tests, interleaving learning, self-explanation, active recalling, to name a few. These learning skills and methodologies enable humans to learn new topics more effectively and efficiently. We are interested in investigating whether humans' learning skills can be borrowed to help machines to learn better. Specifically, we aim to formalize these skills and leverage them to train better machine learning (ML) models. To achieve this goal, we develop a general framework -- Skillearn, which provides a principled way to represent humans' learning skills mathematically and use the formally-represented skills to improve the training of ML models. In two case studies, we apply Skillearn to formalize two learning skills of humans: learning by passing tests and interleaving learning, and use the formalized skills to improve neural architecture search. Experiments on various datasets show that trained using the skills formalized by Skillearn, ML models achieve significantly better performance.</div>

show abstract

“…3.3 1.3 *DropNAS (Hong et al, 2020) 2.58±0.14 4.1 0.6 *FairDARTS (Chu et al, 2019) 2.54 3.3 0.4 *DrNAS 2.54±0.03 4.0 0.4 *DARTS-1st 3.00±0.14 3.3 0.4 SGL-DARTS-1st (ours) 2.41±0.06 3.7 1.2 *DARTS − (Chu et al, 2020a) 2.59±0.08 3.5 0.4 † DARTS − (Chu et al, 2020a) 2.97±0.04 3.3 0.6 SGL-DARTS − (ours) 2.60±0.07 3.1 2.0 *P-DARTS 2.50 3.4 0.3 SGL-P-DARTS (ours) 2.47±0.10 3.6 2.1 *PC-DARTS (Xu et al, 2020) 2.57±0.07 3.6 0.1 SGL-PC-DARTS (ours) 2.60±0.12 3.5 0.5 Table 3: Results on CIFAR-10. * means the results are taken from DARTS − (Chu et al, 2020a), NoisyDARTS (Chu et al, 2020b), and DrNAS .…”

Section: Methodsmentioning

confidence: 99%

Learning by Passing Tests, with Application to Neural Architecture Search

Du¹,

Zhang²,

Xie³

2021

Preprint

View full text Add to dashboard Cite

Learning through tests is a broadly used methodology in human learning and shows great effectiveness in improving learning outcome: a sequence of tests are made with increasing levels of difficulty; the learner takes these tests to identify his/her weak points in learning and continuously addresses these weak points to successfully pass these tests. We are interested in investigating whether this powerful learning technique can be borrowed from human to improve the learning abilities of machines. We propose a novel learning approach called learning by passing tests (LPT). In our approach, a tester model creates increasingly more difficult tests to evaluate a learner model. The learner tries to continuously improve its learning ability so that it can successfully pass however difficult tests created by the tester. We propose a multi-level optimization framework to formulate LPT, where the tester learns to create difficult and meaningful tests and the learner learns to pass these tests. We develop an efficient algorithm to solve the LCT problem. Our method is applied for neural architecture search on and achieves significant improvement over state-of-the-art baselines on CIFAR-100, CIFAR-10, and ImageNet.

show abstract

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Cited by 233 publications

References 15 publications

Learning by Teaching, with Application to Neural Architecture Search

Learning by Teaching, with Application to Neural Architecture Search

Skillearn: Machine Learning Inspired by Humans' Learning Skills

Learning by Passing Tests, with Application to Neural Architecture Search

Contact Info

Product

Resources

About