EPNet: Learning to Exit with Flexible Multi-Branch Network

Abstract

Dynamic inference is an emerging technique that reduces the computational cost of deep neural network under resource-constrained scenarios, such as inference on mobile devices. One way to achieve dynamic inference is to leverage multi-branch neural networks that apply different computation on input data by following different branches. Conventional research on multi-branch neural networks mainly targeted at improving the accuracy of each branch, and use manually designed rules to decide which input follows which branch of the network. Furthermore, these networks often provide a small number of exits, limiting their ability to adapt to external changes. In this paper, we investigate the problem of designing a flexible multi-branch network and early-exiting policies that can adapt to the resource consumption to individual inference request without impacting the inference accuracy. We propose a lightweight branch structure that also provides fine-grained flexibility for earlyexiting and leverage Markov decision process (MDP) to automatically learn the early-exiting policies. Our proposed model, EPNet, was effective in reducing inference cost without impacting accuracy by choosing the most suitable branch exit. We also observe that EPNet achieved 3% higher accuracy with an inference budget, compared to state-of-the-art approaches. CCS CONCEPTS • Information systems → Data mining; • Computing methodologies → Neural networks;

Dynamic Neural Networks: A Survey

“…3 (b)), where early features can be propagated to deep layers if needed. Based on this such architecture design, early exiting can be achieved according to confidence-based criteria [43], [48] or learned decision functions [44], [49], [50], [51]. Note that the confidence-based exiting policy consumes no extra computation during inference, while usually requiring tuning the threshold(s) on the validation set.…”

AdaSparse: Learning Adaptively Sparse Structures for Multi-Domain Click-Through Rate Prediction

“…This technique generally cannot be applied to binary classification and regression tasks. On the other hand, BERxiT [34] and Epnet [6] use learned modules for early-exiting.…”

Adaptive Inference through Early-Exit Networks

“…Expectedly, one may wonder why not to learn network weights and the exit policy jointly. To this direction there has been work approaching the exit policy in differentiable [6,60] and non-differentiable [8] ways. In essence, instead of explicitly measuring the exit's confidence, the decision on whether to exit can be based on the feature maps of the exits themselves.…”

“…Scardapane et al [60] Vision/Classification Differentiable jointly learned exit policy on EE-networks. EpNet [8] Vision/Classification Non-differentiable exit policy for EE-networks. Chen et al [6] Vision/{Classification, Denoising} Jointly learned variational exit policy for EE-networks.…”