Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks

Abstract

We propose to focus on the problem of discovering neural network architectures efficient in terms of both prediction quality and cost. For instance, our approach is able to solve the following tasks: learn a neural network able to predict well in less than 100 milliseconds or learn an efficient model that fits in a 50 Mb memory. Our contribution is a novel family of models called Budgeted Super Networks (BSN). They are learned using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the nature of this cost. We present a set of experiments on computer vision problems and analyze the ability of our technique to deal with three different costs: the computation cost, the memory consumption cost and a distributed computation cost. We particularly show that our model can discover neural network architectures that have a better accuracy than the ResNet and Convolutional Neural Fabrics architectures on CIFAR-10 and CIFAR-100, at a lower cost.

Searching for a Robust Neural Architecture in Four GPU Hours

“…Macro search algorithms aim to directly discover the entire neural networks [5,4,38,46,21]. To search convolutional neural networks (CNNs) [20], typical approaches apply RL to optimize the searching policy to discover architectures [1,5,46,31].…”

Adaptative Inference Cost With Convolutional Neural Mixture Models

“…As discussed, our model allows to dynamically dynamically set the trade-off between accuracy and inference time with no additional cost. ferent state-of-the-art CNN acceleration strategies [17,19,22,38,56]. We consider methods applying pruning at different levels, such as independent filters (Network slimming [38]), groups of weights (CondenseNet) [19], connections in multi-branch architectures (SuperNet) [56], or a combination of them (SSS) [22].…”

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

“…Our work is motivated by differentiable architecture search [34,37,26,1], which is based on the continuous relaxation of the architecture representation, allowing efficient search with gradient descent. [34,37] propose a gridlike network as the search space, while [26] relax the search space to be continuous and search the space by solving a bilevel optimization problem.…”

“…Our work is motivated by differentiable architecture search [34,37,26,1], which is based on the continuous relaxation of the architecture representation, allowing efficient search with gradient descent. [34,37] propose a gridlike network as the search space, while [26] relax the search space to be continuous and search the space by solving a bilevel optimization problem. Other works in architecture search employ reinforcement learning [4,49], evolutionary algorithms [31,39,25], and sequential model-based optimization [29,24] to search the discrete space.…”