The Memory Challenge in Ultra-Low Power Deep Learning

Conti, Francesco; Rusci, Manuele; Benini, Luca

doi:10.1007/978-3-030-18338-7_19

Cited by 4 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The deployment of DL-based algorithms on the IoT demands aggressive hardware, software, and algorithmic co-optimization to exploit the scarce resources on these systems to the maximum degree [6]. In particular, the scarce availability of memory constitutes a real Deep Learning Memory Wall [7]: a fundamental limitation to the maximum performance of an embedded DNN compute system.…”

Section: Introductionmentioning

confidence: 99%

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

Burrello

Garofalo

Bruschi

et al. 2021

IEEE Trans. Comput.

Self Cite

View full text Add to dashboard Cite

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency -requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY ) -an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off-and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5× better MAC/cycle than the GreenWaves proprietary software solution and 18.1× better than the state-of-the-art result on an STM32-H743 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on

show abstract

Section: Introductionmentioning

confidence: 99%

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

Burrello

Garofalo

Bruschi

et al. 2021

IEEE Trans. Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are many reasons for moving computing closer to where sensors gather data, including reliability, limiting network bandwidth, providing improved security, or more effective resource management of deep learning applications. This challenge is gaining research traction, for example, in [4], authors discuss the deep learning memory challenge caused by scarce availability of memory. They propose a combination of techniques for deployment of next-generation of on-chip machine learning including hardware-aware DNNs that could allow for this stage to be on-chip as opposed to offline.…”

Section: Introductionmentioning

confidence: 99%

On-Device Deep Learning Inference for System-on-Chip (SoC) Architectures

Springer¹,

Eiroa-Lledo²,

Stevens³

et al. 2021

Electronics

View full text Add to dashboard Cite

As machine learning becomes ubiquitous, the need to deploy models on real-time, embedded systems will become increasingly critical. This is especially true for deep learning solutions, whose large models pose interesting challenges for target architectures at the “edge” that are resource-constrained. The realization of machine learning, and deep learning, is being driven by the availability of specialized hardware, such as system-on-chip solutions, which provide some alleviation of constraints. Equally important, however, are the operating systems that run on this hardware, and specifically the ability to leverage commercial real-time operating systems which, unlike general purpose operating systems such as Linux, can provide the low-latency, deterministic execution required for embedded, and potentially safety-critical, applications at the edge. Despite this, studies considering the integration of real-time operating systems, specialized hardware, and machine learning/deep learning algorithms remain limited. In particular, better mechanisms for real-time scheduling in the context of machine learning applications will prove to be critical as these technologies move to the edge. In order to address some of these challenges, we present a resource management framework designed to provide a dynamic on-device approach to the allocation and scheduling of limited resources in a real-time processing environment. These types of mechanisms are necessary to support the deterministic behavior required by the control components contained in the edge nodes. To validate the effectiveness of our approach, we applied rigorous schedulability analysis to a large set of randomly generated simulated task sets and then verified the most time critical applications, such as the control tasks which maintained low-latency deterministic behavior even during off-nominal conditions. The practicality of our scheduling framework was demonstrated by integrating it into a commercial real-time operating system (VxWorks) then running a typical deep learning image processing application to perform simple object detection. The results indicate that our proposed resource management framework can be leveraged to facilitate integration of machine learning algorithms with real-time operating systems and embedded platforms, including widely-used, industry-standard real-time operating systems.

show abstract

A Conceptual Framework for Stochastic Neuromorphic Computing

Gaines

2021

IEEE Des. Test

View full text Add to dashboard Cite

The Memory Challenge in Ultra-Low Power Deep Learning

Cited by 4 publications

References 34 publications

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

On-Device Deep Learning Inference for System-on-Chip (SoC) Architectures

A Conceptual Framework for Stochastic Neuromorphic Computing

Contact Info

Product

Resources

About