On-Device Deep Learning Inference for System-on-Chip (SoC) Architectures

Abstract

As machine learning becomes ubiquitous, the need to deploy models on real-time, embedded systems will become increasingly critical. This is especially true for deep learning solutions, whose large models pose interesting challenges for target architectures at the “edge” that are resource-constrained. The realization of machine learning, and deep learning, is being driven by the availability of specialized hardware, such as system-on-chip solutions, which provide some alleviation of constraints. Equally important, however, are the operating systems that run on this hardware, and specifically the ability to leverage commercial real-time operating systems which, unlike general purpose operating systems such as Linux, can provide the low-latency, deterministic execution required for embedded, and potentially safety-critical, applications at the edge. Despite this, studies considering the integration of real-time operating systems, specialized hardware, and machine learning/deep learning algorithms remain limited. In particular, better mechanisms for real-time scheduling in the context of machine learning applications will prove to be critical as these technologies move to the edge. In order to address some of these challenges, we present a resource management framework designed to provide a dynamic on-device approach to the allocation and scheduling of limited resources in a real-time processing environment. These types of mechanisms are necessary to support the deterministic behavior required by the control components contained in the edge nodes. To validate the effectiveness of our approach, we applied rigorous schedulability analysis to a large set of randomly generated simulated task sets and then verified the most time critical applications, such as the control tasks which maintained low-latency deterministic behavior even during off-nominal conditions. The practicality of our scheduling framework was demonstrated by integrating it into a commercial real-time operating system (VxWorks) then running a typical deep learning image processing application to perform simple object detection. The results indicate that our proposed resource management framework can be leveraged to facilitate integration of machine learning algorithms with real-time operating systems and embedded platforms, including widely-used, industry-standard real-time operating systems.

Deep Neural Network: an Alternative to Traditional Channel Estimators in Massive MIMO Systems

“…DNNs are structures that can be massively parallelized. This property has been exploited through the construction of a specific System on Chip (SoC) [37] that can execute GFLOPS and TFLOPS [38] keeping controlled both the consumption and the clock frequency necessary for the operation of the chip. In terms of system scalability, the neural models implemented are trained by focusing on their ability to generalize, seeking to maintain its quality under different scenarios making a massive deployment more transparent.…”

Semantic Segmentation Network Slimming and Edge Deployment for Real-Time Forest Fire or Flood Monitoring Systems Using Unmanned Aerial Vehicles

“…Lastly, a DSP is an auxiliary processor specialized in processing numerical operations at high speed and repeatedly performing multiply-accumulate operations [3], so it is appropriate for accelerating the inference of DNNs in edge devices. Usually, a DSP is mounted on a system-on-chip (SoC) along with a CPU and a GPU [4].…”