Model slicing for supporting complex analytics with elastic inference cost and resource constraints

Abstract

Deep learning models have been used to support analytics beyond simple aggregation, where deeper and wider models have been shown to yield great results. These models consume a huge amount of memory and computational operations. However, most of the large-scale industrial applications are often computational budget constrained. In practice, the peak workload of inference service could be 10x higher than the average cases, with the presence of unpredictable extreme cases. Lots of computational resources could be wasted during off-peak hours and the system may crash when the workload exceeds system capacity. How to support deep learning services with dynamic workload cost-efficiently remains a challenging problem. In this paper, we address the challenge with a general and novel training scheme called model slicing , which enables deep learning models to provide predictions within the prescribed computational resource budget dynamically. Model slicing could be viewed as an elastic computation solution without requiring more computational resources. Succinctly, each layer in the model is divided into groups of contiguous block of basic components (i.e. neurons in dense layers and channels in convolutional layers), and then partially ordered relation is introduced to these groups by enforcing that groups participated in each forward pass always starts from the first group to the dynamically-determined rightmost group. Trained by dynamically indexing the rightmost group with a single parameter slice rate , the network is engendered to build up group-wise and residual representation. Then during inference, a sub-model with fewer groups can be readily deployed for efficiency whose computation is roughly quadratic to the width controlled by the slice rate. Extensive experiments show that models trained with model slicing can effectively support on-demand workload with elastic inference cost.

SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis

“…Model slicing [5] is a general technique to enable deep learning models to support elastic computation. Specifically, each layer of the model is divided into equal-sized contiguous computational groups.…”

“…As a result, during inference, we can support accuracy-efficiency trade-offs by dynamically slicing a subnet of a certain width, where only the parameters of the activated groups are involved in computation. Theoretically, the number of parameters and computation measured in FLOPs are both roughly quadratic to the slice rate 𝑟 [5], e.g., a slice rate of 0.5 can achieve up to four times speedup. Therefore, we can support elastic inference by introducing the model slicing technique to the training stage.…”

“…VGG [39], ResNet [16] Image Classification Mask RCNN [15] Object Detection Deep Speech [2] Speech Recognition BERT [10] Question and Answering Random Forest [17] Tabular Classification XGBoost [7] Tabular Regression 5.1.1 Datasets. We evaluate the training and inference efficiency of SINGA-Easy using model ResNet-50 on three image classification datasets, namely CIFAR [23], ILSVRC 2012 [9] and NIH Chest X-rays dataset 5 . To demonstrate the general support of SINGA-Easy for different applications, we train Inception-ResNet-v2 and yoloV3 [35] on five food datasets for food image classification and detection and visualize the results.…”

Dynamic slicing for deep neural networks

“…Shen et al [61] proposed slicing CNN feature maps to understand the appearance and dynamic features in the videos. Cai et al [9] proposed to slice a DNN into different groups that can be assembled elastically to support dynamic workload. However, these approaches are not related to program slicing that aims to understand the internal logic of a program.…”

LoCEC: Local Community-based Edge Classification in Large Online Social Networks

“…In this work, we first carefully analyze how different types of user pairs interact on the WeChat platform and find useful interaction features for identifying relationship types. And then we summarize two major challenges in the edge classification task: The data sparsity problem [15] and the computation bottleneck in networks with billions of nodes [16].…”