MOSAIC: Heterogeneity-, Communication-, and Constraint-Aware Model Slicing and Execution for Accurate and Efficient Inference

Han, Myeonggyun; Hyun, Jihoon; Park, Seongbeom; Park, Jinsu; Baek, Woongki

doi:10.1109/pact.2019.00021

Cited by 25 publications

(23 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To investigate the deep learning training workload, we profiled the latency of each step in the network model training. In FP, BP, and UP, we confirmed that the latency of each step is different depending on the characteristics of the layer, the size of the filter, and the size of the input data [ 22 , 24 , 25 ]. We profiled the execution time of the CPU and GPU while training the ResNet-18 model, as depicted in Figure 6 .…”

Section: Methodsmentioning

confidence: 79%

“…Considering the effective use of the available computing power of mobile devices, some reports propose methods to execute each layer of the network on different processors. DeepX [ 21 ] and Mosaic [ 25 ] distribute the execution of model layers to different computing resources such as the CPU, GPU, DSP, or neural processing unit (NPU). In contrast,

layer [ 22 ] proposed the execution of a single neural network layer using both the CPU and GPU.…”

Section: Related Workmentioning

confidence: 99%

“…In contrast,

layer [ 22 ] proposed the execution of a single neural network layer using both the CPU and GPU. All the approaches used in previous works [ 18 , 19 , 20 , 21 , 22 , 25 , 26 ] are limited in that they only focus on accelerating the inference by optimizing the neural network model or efficiently using multiple processors of mobile devices.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, existing studies applying deep learning algorithms to mobile devices mainly focus on accelerating the deep learning inference, which requires relatively low computing power. To accelerate the deep learning inference, existing approaches optimize inference for mobile processors [ 18 , 19 , 20 ] or perform inference by dividing the model across multiple computing resources of the mobile device [ 18 , 21 , 22 , 23 , 24 , 25 ]. Other approaches focus on enhancing the usability of memory to eliminate data copy time [ 22 , 23 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Accelerating On-Device Learning with Layer-Wise Processor Selection Method on Unified Memory

Kim

Moon

et al. 2021

Sensors

View full text Add to dashboard Cite

Recent studies have applied the superior performance of deep learning to mobile devices, and these studies have enabled the running of the deep learning model on a mobile device with limited computing power. However, there is performance degradation of the deep learning model when it is deployed in mobile devices, due to the different sensors of each device. To solve this issue, it is necessary to train a network model specific to each mobile device. Therefore, herein, we propose an acceleration method for on-device learning to mitigate the device heterogeneity. The proposed method efficiently utilizes unified memory for reducing the latency of data transfer during network model training. In addition, we propose the layer-wise processor selection method to consider the latency generated by the difference in the processor performing the forward propagation step and the backpropagation step in the same layer. The experiments were performed on an ODROID-XU4 with the ResNet-18 model, and the experimental results indicate that the proposed method reduces the latency by at most 28.4% compared to the central processing unit (CPU) and at most 21.8% compared to the graphics processing unit (GPU). Through experiments using various batch sizes to measure the average power consumption, we confirmed that device heterogeneity is alleviated by performing on-device learning using the proposed method.

show abstract

Section: Methodsmentioning

confidence: 79%

layer [ 22 ] proposed the execution of a single neural network layer using both the CPU and GPU.…”

Section: Related Workmentioning

confidence: 99%

“…In contrast,

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Accelerating On-Device Learning with Layer-Wise Processor Selection Method on Unified Memory

Kim

Moon

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The community has even started declaring victory in achieving human-level accuracies for certain ML-based tasks (e.g., image recognition and classification) [25,26]. Built upon such outstanding advances, the industry is now moving toward integrating the ML algorithms into various types of real-world applications and deploying the applications on the edge platforms [4,22,23,29,33,35,38,47,63,65]. In many real-world scenarios, the edge platforms often serve multiple purposes and have to handle different types of inference requests for different ML models at the same time.…”

Section: Inference In Edge Platformsmentioning

confidence: 99%

SLO-Aware Inference Scheduler for Heterogeneous Processors in Edge Platforms

Seo

Cha

Kim

et al. 2021

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

With the proliferation of applications with machine learning (ML), the importance of edge platforms has been growing to process streaming sensor, data locally without resorting to remote servers. Such edge platforms are commonly equipped with heterogeneous computing processors such as GPU, DSP, and other accelerators, but their computational and energy budget are severely constrained compared to the data center servers. However, as an edge platform must perform the processing of multiple machine learning models concurrently for multimodal sensor data, its scheduling problem poses a new challenge to map heterogeneous machine learning computation to heterogeneous computing processors. Furthermore, processing of each input must provide a certain level of bounded response latency, making the scheduling decision critical for the edge platform. This article proposes a set of new heterogeneity-aware ML inference scheduling policies for edge platforms. Based on the regularity of computation in common ML tasks, the scheduler uses the pre-profiled behavior of each ML model and routes requests to the most appropriate processors. It also aims to satisfy the service-level objective (SLO) requirement while reducing the energy consumption for each request. For such SLO supports, the challenge of ML computation on GPUs and DSP is its inflexible preemption capability. To avoid the delay caused by a long task, the proposed scheduler decomposes a large ML task to sub-tasks by its layer in the DNN model.

show abstract

Joint heterogeneity-aware personalized federated search for energy efficient battery-powered edge computing

Zhao

Zhang

et al. 2023

Future Generation Computer Systems

View full text Add to dashboard Cite

MOSAIC: Heterogeneity-, Communication-, and Constraint-Aware Model Slicing and Execution for Accurate and Efficient Inference

Cited by 25 publications

References 37 publications

Accelerating On-Device Learning with Layer-Wise Processor Selection Method on Unified Memory

Accelerating On-Device Learning with Layer-Wise Processor Selection Method on Unified Memory

SLO-Aware Inference Scheduler for Heterogeneous Processors in Edge Platforms

Joint heterogeneity-aware personalized federated search for energy efficient battery-powered edge computing

Contact Info

Product

Resources

About