Learning by Passing Tests, with Application to Neural Architecture Search

Abstract

Learning through tests is a broadly used methodology in human learning and shows great effectiveness in improving learning outcome: a sequence of tests are made with increasing levels of difficulty; the learner takes these tests to identify his/her weak points in learning and continuously addresses these weak points to successfully pass these tests. We are interested in investigating whether this powerful learning technique can be borrowed from human to improve the learning abilities of machines. We propose a novel learning approach called learning by passing tests (LPT). In our approach, a tester model creates increasingly more difficult tests to evaluate a learner model. The learner tries to continuously improve its learning ability so that it can successfully pass however difficult tests created by the tester. We propose a multi-level optimization framework to formulate LPT, where the tester learns to create difficult and meaningful tests and the learner learns to pass these tests. We develop an efficient algorithm to solve the LCT problem. Our method is applied for neural architecture search on and achieves significant improvement over state-of-the-art baselines on CIFAR-100, CIFAR-10, and ImageNet.

Neural Architecture Search For Skin Cancer Detection

“…Learning By Passing Tests 3 Studies show that human learning is constantly enhanced when paired with evaluation in the form of tests. The more appropriate the tests are to the content that is being delivered, there is a direct proportion in terms of effective learning.…”

“…This procedure is along the lines of a bilevel technique. Studies have been done recently 3 , where this approach is applied in the learning of machines, particularly in the task of Neural Architecture Search. It uses two models, a learner that learns to carry out the architecture search and a tester that aims at learning to test the architecture searched more strictly.…”

Learning by Examples Based on Multi-level Optimization

“…BLO has been extended to multi-level optimization (MLO) which involves more than two levels of optimization problems. MLO has been applied for data generation [58], interleaving multi-task learning [2], data reweighting in domain adaptation [68], explainable learning [23], humaninspired learning [66], curriculum evaluation [13], mutual knowledge distillation [12], end-to-end knowledge distillation [55], etc.…”