FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things

Abstract

The growing number of low-power smart devices in the Internet of Things is coupled with the concept of "Edge Computing", that is moving some of the intelligence, especially machine learning, towards the edge of the network. Enabling machine learning algorithms to run on resource-constrained hardware, typically on low-power smart devices, is challenging in terms of hardware (optimized and energy-efficient integrated circuits), algorithmic and firmware implementations. This paper presents FANN-on-MCU, an open-source toolkit built upon the Fast Artificial Neural Network (FANN) library to run lightweight and energy-efficient neural networks on microcontrollers based on both the ARM Cortex-M series and the novel RISC-Vbased Parallel Ultra-Low-Power (PULP) platform. The toolkit takes multi-layer perceptrons trained with FANN and generates code targeted to low-power microcontrollers. This paper also presents detailed analyses of energy efficiency across the different cores, and the optimizations to handle different network sizes. Moreover, it provides a detailed analysis of parallel speedups and degradations due to parallelization overhead and memory transfers. Further evaluations include experimental results for three different applications using a self-sustainable wearable multi-sensor bracelet. Experimental results show a measured latency in the order of only a few microseconds and power consumption of a few milliwatts while keeping the memory requirements below the limitations of the targeted microcontrollers. In particular, the parallel implementation on the octa-core RISC-V platform reaches a speedup of 22x and a 69% reduction in energy consumption with respect to a single-core implementation on Cortex-M4 for continuous real-time classification.

Q-EEGNet: an Energy-Efficient 8-bit Quantized Parallel EEGNet Implementation for Edge Motor-Imagery Brain-Machine Interfaces

“…However, the resulting implementation for RISC-V does not support parallel execution. FANN-ON-MCU [12] is a different framework for exporting optimized neural networks to ARM processors, and to PULP-based systems. However, this framework does not offer convolutional layers required for EEGNET.…”

“…Recently, researchers have developed the parallel ultra-low power (PULP) platform based on the RISC-V Instruction Set Architecture (ISA) [9], [10] , which is built around the concept of using simple cores for energy efficiency, while recovering and scaling up performance through parallelism. PULP MCUs have proven to outperform the Cortex-M family by at least one order of magnitude in energy efficiency [11], [12]. In particular, Mr. Wolf, with its 8-core compute cluster and custom ISA extensions, can reach up to 274 GOp/s/W [13].…”

EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain–Machine Interfaces

“…Processing the data near the sensor on a low-power microcontroller unit (MCU) allows us to mitigate these concerns. However, accurate networks such as the TPCT model have 7.78 M trainable parameters and require 1.73 billion multiply-accumulate (MAC) operations per inference, which is out of reach of a typical low-power MCU with few MB of Flash and few hundreds of kB of RAM [18]. Alternatively, more compact models such as EEGNet with 2.5 k parameters and 13 MMACs can come to the rescue and have been successfully implemented on MCUs [19], [20].…”

An Accurate EEGNet-based Motor-Imagery Brain–Computer Interface for Low-Power Edge Computing

“…Recently, a new generation of wearable BCIs is attracting the academic and industrial researchers. An increasing number of battery-operated wearable solutions, using microcontroller units (MCUs), are proposed to bring computing capabilities towards the "edge" to perform real-time near-sensor computation [10], [12], [13], [14]. Edge computing and near-sensor computation offer the following advantages: 1) lower energy consumption for the data transmission between sensors and remote processing; 2) longer battery lifetime; 3) significantly shorter latency compared to remote computation; 4) user comfort; 5) security and privacy improvements, as the data are processed locally and only little information is transmitted wirelessly if necessary.…”