MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

Abstract

Most existing deep neural networks are static, which means they can only do inference at a fixed complexity. But the resource budget can vary substantially across different devices. Even on a single device, the affordable budget can change with different scenarios, and repeatedly training networks for each required budget would be incredibly expensive. Therefore, in this work, we propose a general method called MutualNet to train a single network that can run at a diverse set of resource constraints. Our method trains a cohort of model configurations with various network widths and input resolutions. This mutual learning scheme not only allows the model to run at different width-resolution configurations but also transfers the unique knowledge among these configurations, helping the model to learn stronger representations overall. MutualNet is a general training methodology that can be applied to various network structures (e.g., 2D networks: MobileNets, ResNet, 3D networks: SlowFast, X3D) and various tasks (e.g., image classification, object detection, segmentation, and action recognition), and is demonstrated to achieve consistent improvements on a variety of datasets. Since we only train the model once, it also greatly reduces the training cost compared to independently training several models. Surprisingly, MutualNet can also be used to significantly boost the performance of a single network, if dynamic resource constraint is not a concern. In summary, MutualNet is a unified method for both static and adaptive, 2D and 3D networks. Codes and pre-trained models are available at https://github.com/taoyang1122/MutualNet.

Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms

“…When a single dynamic-OFA runs on the device, a look-up table is used to directly find the sub-network 'level' to meet different user-defined accuracy and latency constraints. This is a similar approach to previous dynamic DNNs like Slimmable [23][24][25] and Mu-tualNet [21]. 2.…”

“…These models can run different active channels and achieve instant and adaptive accuracy-latency trade-offs. MutualNet [21] shows improved performance by adding input resolutions with width as switchable dimensions. For different constraints, different sub-networks with varying widths and resolutions can be chosen and built as a query table.…”

“…Dynamic-OFA is developed for both the CPU and GPU of the Nvidia Jetson Xavier NX platform, and the accuracy and latency are empirically measured. The results are compared with both state-of-the-art static OFA backbone [2] and dynamic DNNs [21,[23][24][25]. We have also executed other applications (e.g.…”

“…Since both software performance requirements and hardware resource availability can change dynamically at runtime [19,20], various dynamic DNNs [21,[23][24][25] have been proposed to address this issue. These dynamic DNNs contain various sub-networks that each have a different accuracy and latency.…”

“…The total training would be costly in real deployment scenarios, since platform-aware pruning/NAS need to be applied before dynamic DNN process, and all model variants of different deployment scenarios have to be retrained to become dynamic. Second, prior works rescale the network architecture to obtain sub-networks by either channel scaling [21,[23][24][25] or layer scaling [9]. However, the most efficient DNN architectures on CPUs typically have more layers but fewer channels, while the oppo-site is true for GPUs [3].…”