Sound event detection with binary neural networks on tightly power-constrained IoT devices

Abstract

Sound event detection (SED) is a hot topic in consumer and smart city applications. Existing approaches based on deep neural networks (DNNs) are very effective, but highly demanding in terms of memory, power, and throughput when targeting ultra-low power always-on devices.Latency, availability, cost, and privacy requirements are pushing recent IoT systems to process the data on the node, close to the sensor, with a very limited energy supply, and tight constraints on the memory size and processing capabilities precluding to run state-of-the-art DNNs.In this paper, we explore the combination of extreme quantization to a small-footprint binary neural network (BNN) with the highly energy-efficient, RISC-V-based (8+1)-core GAP8 microcontroller. Starting from an existing CNN for SED whose footprint (815 kB) exceeds the 512 kB of memory available on our platform, we retrain the network using binary filters and activations to match these memory constraints. (Fully) binary neural networks come with a natural drop in accuracy of 12-18% on the challenging Ima-geNet object recognition challenge compared to their equivalent full-precision baselines. This BNN reaches a 77.9% accuracy, just 7% lower than the full-precision version, with 58 kB (7.2× less) for the weights and 262 kB (2.4× less) memory in total. With our BNN implementation, we reach a peak throughput of 4.6 GMAC/s and 1.5 GMAC/s over the full network, including preprocessing with Mel bins, which corresponds to an efficiency of 67.1 GMAC/s/W and 31.3 GMAC/s/W, respectively. Compared to the performance of an ARM Cortex-M4 implementation, our system has a 10.3× faster execution time and a 51.1× higher energy-efficiency.

Pruning vs XNOR-Net: A Comprehensive Study of Deep Learning for Audio Classification on Edge-Devices

“…The current state-of-the-art XNOR models for the benchmark image datasets are: [23] for MNIST, [24] for CIFAR-10 and CIFAR-100 and [25] for ImageNet. In contrast, we have found [26] to be the only XNOR-Net based work for audio classification. However, that work uses spectrograms as the input to the model which is similar to image classification using XNOR-Net.…”

“…To the best of our knowledge, Cerutti et al [26] have presented the only XNOR network for audio classification so far, termed BNN-GAP8. They have used a different benchmark, namely, the less commonly used AudioEvent dataset [43].…”

“…There are many state-of-the-art XNOR-Net models for different computer vision tasks, such as Rastegari et al [23] for MNIST, Cong [24] for CIFAR-10, and Bulat et al [25] for CIFAR-100 and ImageNet datasets. However, for audio classification, the only work we are aware of is presented by Cerutti et al [26]. This study uses XNOR-Net in combination with spectrogram-based input to effectively perform audio classification via image classification.…”

CUTIE: Beyond PetaOp/s/W Ternary DNN Inference Acceleration With Better-Than-Binary Energy Efficiency

“…It has been known for some time that quantization of network weights to 5 bits and less is possible without a loss in accuracy in comparison to a 32-bit floating-point baseline model [5], [6], [7]. Further quantization of network weights to binary or ternary precision usually results in a small drop in accuracy, but precision is still adequate for many applications [12], [13], [29], [30]. Extending the approach of extreme quantization to intermediate activations, fully binarized and fully ternarized networks have been proposed [9], [15].…”

ChewBaccaNN: A Flexible 223 TOPS/W BNN Accelerator

“…The first two BNNs have been used in embedded applications. Cerutti et al presented a BNN for Sound Event Detection on the Freesound database with 28 classes [26]. The audio data is converted to a Mel-frequency cepstral spectrogram and fed to a binary CNN with 5 layers with 3×3 kernels, followed by 3 layers with 1×1 kernels.…”