Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training

Abstract

Much recent research has been dedicated to improving the efficiency of training and inference for image classification. This effort has commonly focused on explicitly improving theoretical efficiency, often measured as ImageNet validation accuracy per FLOP. These theoretical savings have, however, proven challenging to achieve in practice, particularly on high-performance training accelerators.In this work, we focus on improving the practical efficiency of the state-of-the-art EfficientNet models on a new class of accelerator, the Graphcore IPU. We do this by extending this family of models in the following ways: (i) generalising depthwise convolutions to group convolutions; (ii) adding proxy-normalized activations to match batch normalization performance with batch-independent statistics; (iii) reducing compute by lowering the training resolution and inexpensively fine-tuning at higher resolution. We find that these three methods improve the practical efficiency for both training and inference. Our code will be made available online.

Evaluating Robustness to Noise and Compression of Deep Neural Networks for Keyword Spotting

“…Although the number of parameters of the EfficientNet is 6.87 times greater than the SqueezeNet, the inference time for both models is similar. This is expected since the models of the EfficientNet family are designed to focus on increasing accuracy performance while minimizing the overall number of operations required [15], [59]. Considering file size and accuracy as decision-making metrics for the choice of deep keyword spotting models, among the selected one, SqueezeNet can be considered the best compromise model.…”