DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

Abstract

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency -requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY ) -an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off-and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5× better MAC/cycle than the GreenWaves proprietary software solution and 18.1× better than the state-of-the-art result on an STM32-H743 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on

Fully Onboard AI-Powered Human-Drone Pose Estimation on Ultralow-Power Autonomous Flying Nano-UAVs

“…: Weight [cm:kg] Power [W] Onboard device standard-size [4] ∼ 50 : ≥ 1 ≥ 100 Desktop micro-size [5] ∼ 25 : ∼ 0.5 ∼ 50 Embedded nano-size [14] ∼ 10 : ∼ 0.01 ∼ 5 MCU pico-size [13] ∼ 2 : ≤ 0.001 ∼ 0.1 ULP both the strict power budget of IoT MCUs and the realtime requirement of autonomous nano-drones; • we present our dataset augmentation methodology, which maximizes the model's generalization capability with synthetic pitch, photometric, optical, and geometric enhancements; • using open-source tools [19], [20], we demonstrate our methodology from perception to control (including training, aggressive 8-bit quantization, CNN deployment, and low-level controller), with no drop in regression performance, even compared to the full precision (float 32-bit) Proximity CNN. We achieve an onboard peak inference performance of 135 frame/s within 86 mW and a top energy efficiency of ∼0.43 mJ/frame; • we experimentally evaluate how the CNN design impacts on i) regression performance, ii) power consumption, iii) inference rate, and iv) closed-loop control accuracy; • we prove our methodology in the field presenting a closed-loop, fully working demonstration of PULP-Frontnet on a 27-grams nano-UAVs, achieving 100% success-rate on all tests (18 runs on never-seen-before subjects), with behavior comparable with an ideal motioncapture system (median absolute angular error below 5 • );…”

“…• making use of open-source quantization/deployment tools [19], [20], as well as employing a 2× more aggressive quantization scheme (i.e., 8-bits vs. 16-bits); • including the development flow for ad-hoc dataset collection and its augmentation; • proposing a novel streamlined DL model (up to 10× and 8× fewer operations and memory, respectively); • introducing a thorough model-size analysis to study the relation between power consumption, memory constraints, regression performance, and control accuracy. Ultimately, our models push further the onboard NN's inference performance with a peak throughput of 135 frame/s @ 86 mW -PULP-Dronet peaked at 18 frame/s @ 272 mW.…”

A Multi-Core Controller for an Embedded AI System Supporting Parallel Recognition

“…Unlike the research of [ 21 , 23 ], some studies focus on optimizing the software deep learning algorithm to fit existing embedded system-on-chip (SoC) [ 25 , 26 , 27 ]. Adopting the deep learning algorithm with applicable performance to the embedded SoC is extremely hard because of the critically limited memory and storage resources compared to cloud AI or mobile AI devices [ 25 , 26 , 27 ]. In order to fix these problems, the authors of [ 25 , 27 ] proposed frameworks for optimized neural network generation.…”

“…Adopting the deep learning algorithm with applicable performance to the embedded SoC is extremely hard because of the critically limited memory and storage resources compared to cloud AI or mobile AI devices [ 25 , 26 , 27 ]. In order to fix these problems, the authors of [ 25 , 27 ] proposed frameworks for optimized neural network generation. Both frameworks provide quantization of floating-point arithmetic to integer arithmetic and applying memory constants for scaling the neural network for each device.…”

“…As a result, adopting the framework results in a remarkable reduction in latency and maximum SRAM usage. The framework in [ 27 ] concentrates on software optimization, which can be achieved by reflecting the configuration of mirror layered memory architecture and DMA provided by the target hardware. The result shows that the memory transferring overhead caused by cache memory is almost hidden.…”