DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing From Decentralized Data

Abstract

Deep neural speech and audio processing systems have a large number of trainable parameters, a relatively complex architecture, and require a vast amount of training data and computational power. These constraints make it more challenging to integrate such systems into embedded devices and utilize them for real-time, real-world applications. We tackle these limitations by introducing DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition using pre-trained image Convolutional Neural Networks (CNNs). The framework creates and augments Mel spectrogram plots on the fly from raw audio signals which are then used to finetune specific pre-trained CNNs for the target classification task. Subsequently, the whole pipeline can be run in real-time with a mean inference lag of 242.0 ms when a DenseNet121 model is used on a consumer-grade Motorola moto e7 plus smartphone. DeepSpectrumLite operates decentralized, eliminating the need for data upload for further processing. We demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing by obtaining state-of-the-art results on a set of paralinguistic and general audio tasks, including speech and music emotion recognition, social signal processing, COVID-19 cough and COVID-19 speech analysis, and snore sound classification. We provide an extensive command-line interface for users and developers which is comprehensively documented and publicly available at https://github.com/DeepSpectrum/DeepSpectrumLite.

HEAR4Health: a blueprint for making computer audition a staple of modern healthcare

“…There have consequently been increasing efforts to develop AutoML approaches that optimise a large network until it is executable on a low-resource device ( 77 , 78 ). Many of these approaches focus on reducing the memory footprint and the computational complexity of a network while preserving its accuracy.…”

The ACM Multimedia 2022 Computational Paralinguistics Challenge

“…It has been used in previous challenges [31,32] and is described in [3]. A lightweight version of DeepSpectrum for audio signal processing on-device can be found in [5] 5 . auDeep: 6 This feature set is obtained through unsupervised representation learning with recurrent sequence-to-sequence autoencoders [2,11]; it has as well been employed in previous challenges [31,32].…”

Chronic cough—the limitation and advances in assessment techniques

“…Mobile cough devices are portable, low-cost, real-time, and patient-friendly. Thus, in recent years, there has been an increased focus on the advanced techniques based on mobile technologies and wearable devices in the acquisition and automatic processing of cough sounds, especially after the outbreak of coronavirus disease 2019 (COVID-19), more studies have attempted to rapidly identify COVID-19 by this technique (158)(159)(160)(161)(162)(163)(164)(165)(166). Hoyos-Barceló et al proposed an efficient and power-saving smartphone-based cough detection system, which could classify cough in a noisy environment with 88.94% sensitivity and 98.64% specificity (167).…”