DjiNN and Tonic

Abstract

As applications such as Apple Siri, Google Now, Microsoft Cortana, and Amazon Echo continue to gain traction, webservice companies are adopting large deep neural networks (DNN) for machine learning challenges such as image processing, speech recognition, natural language processing, among others. A number of open questions arise as to the design of a server platform specialized for DNN and how modern warehouse scale computers (WSCs) should be outfitted to provide DNN as a service for these applications.In this paper, we present DjiNN, an open infrastructure for DNN as a service in WSCs, and Tonic Suite, a suite of 7 endto-end applications that span image, speech, and language processing. We use DjiNN to design a high throughput DNN system based on massive GPU server designs and provide insights as to the varying characteristics across applications. After studying the throughput, bandwidth, and power properties of DjiNN and Tonic Suite, we investigate several design points for future WSC architectures. We investigate the total cost of ownership implications of having a WSC with a disaggregated GPU pool versus a WSC composed of homogeneous integrated GPU servers. We improve DNN throughput by over 120⇥ for all but one application (40⇥ for Facial Recognition) on an NVIDIA K40 GPU. On a GPU server composed of 8 NVIDIA K40s, we achieve near-linear scaling (around 1000⇥ throughput improvement) for 3 of the 7 applications. Through our analysis, we also find that GPU-enabled WSCs improve total cost of ownership over CPU-only designs by 4-20⇥, depending on the composition of the workload.

The Architectural Implications of Autonomous Driving

“…There is also a large body of work accelerating machine learning-based applications using various accelerator platforms [1, 7ś9, 12ś14, 19, 23ś25, 30, 31, 36, 38, 41, 42, 58, 63, 80]. Speciically, GPUs have been shown to ofer orders of magnitude performance improvement over multicore CPUs [24,25,27,58]. This is because many machine learning algorithms spend a large fraction of their execution time performing matrix multiplication, which can be parallelized on the large number of threads ofered by GPUs.…”

WSMeter

“…To improve a WSC's performance and cost-effectiveness, a large volume of studies discuss resource-and performanceaware job scheduling [18-21, 36-38, 44, 49, 52-54, 56], adaptive power management [29,31,39,40,42], novel architectures [9,10,27], and systematic performance investigation methods [2,17,22,25,30,33,47,48,50]. Unfortunately, to the best of our knowledge, there is no standard methodology to evaluate the holistic performance of a WSC running thousands of distinct jobs.…”

An intelligent collaborative inference approach of service partitioning and task offloading for deep learning based service in mobile edge computing networks

“…In order to achieve high‐efficiency inference in these immersive services on mobile device, we can offload some computation tasks to effectively leverage computing resources in edge servers and cloud servers. Coincidentally, we find that the deep neural network (DNN) is the most common of ML techniques 12 and the DNN model can be split into different portions on layer‐level partition 13 . Therein, partial offloading can sometimes outperform binary offloading because an internal layer inside the DNN model usually yields a smaller intermediate output than input layer.…”