Low-resource Multi-task Audio Sensing for Mobile and Embedded Devices via Shared Deep Neural Network Representations

Abstract

Continuous audio analysis from embedded and mobile devices is an increasingly important application domain. More and more, appliances like the Amazon Echo, along with smartphones and watches, and even research prototypes seek to perform multiple discriminative tasks simultaneously from ambient audio; for example, monitoring background sound classes (e.g., music or conversation), recognizing certain keywords ('Hey Siri' or 'Alexa'), or identifying the user and her emotion from speech. The use of deep learning algorithms typically provides state-of-the-art model performances for such general audio tasks. However, the large computational demands of deep learning models are at odds with the limited processing, energy and memory resources of mobile, embedded and IoT devices. In this paper, we propose and evaluate a novel deep learning modeling and optimization framework that specifically targets this category of embedded audio sensing tasks. Although the supported tasks are simpler than the task of speech recognition, this framework aims at maintaining accuracies in predictions while minimizing the overall processor resource footprint. The proposed model is grounded in multi-task learning principles to train shared deep layers and exploits, as input layer, only statistical summaries of audio filter banks to further lower computations. We find that for embedded audio sensing tasks our framework is able to maintain similar accuracies, which are observed in comparable deep architectures that use single-task learning and typically more complex input layers. Most importantly, on an average, this approach provides almost a 2.1× reduction in runtime, energy, and memory for four separate audio sensing tasks, assuming a variety of task combinations. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing systems and tools; • Computer systems organization → Embedded systems;

Multi-task Self-Supervised Learning for Human Activity Detection

“…Over the last years, deep neural networks have been widely adopted for time-series and sensory data processing; achieving impressive performance in several application areas pertaining to pervasive sensing, ubiquitous computing, industries, health and well-being [17,21,38,56,60,73]. In particular, for smartphone-based human Fig.…”

Deep Learning in Mobile and Wireless Networking: A Survey

“…Therefore, a single model can be trained to fulfill multiple objectives, without requiring complete model retraining for different tasks. We argue that this is essential for mobile network engineering, as it reduces computational and memory requirements of mobile systems when performing multitask learning applications [97].…”

“…Analysis [17], [112], [235]- [266] [73], [97], [187], [267]- [291] Mobility Analysis [227], [292]- [310] User Localization [272], [273], [311]- [315] [111], [316]- [334] Wireless Sensor Networks [335]- [346], [346]- [356] Network Control [186], [293], [357]- [368] [234], [368]- [403] Network Security [185], [345], [404]- [419] [223], [420]- [429], [429]- [436] Signal Processing [378], [380], [437]- [444] [322], [445]- [458] Emerging Applications For each domain, we summarize work broadly in tabular form, providing readers with a general picture of individual topics. Most important works in each domain are discussed in more details in text.…”

“…In this section, we look at how to tailor deep learning to mobile networking applications from three perspectives, namely, mobile devices and systems, distributed data centers, and changing mobile network environments. [513] Filter size shrinking, reducing input channels and late downsampling CNN Howard et al [514] Depth-wise separable convolution CNN Zhang et al [515] Point-wise group convolution and channel shuffle CNN Zhang et al [516] Tucker decomposition AE Cao et al [517] Data parallelization by RenderScript RNN Chen et al [518] Space exploration for data reusability and kernel redundancy removal CNN Rallapalli et al [519] Memory optimizations CNN Lane et al [520] Runtime layer compression and deep architecture decomposition MLP, CNN Huynh et al [521] Caching, Tucker decomposition and computation offloading CNN Wu et al [522] Parameters quantization CNN Bhattacharya and Lane [523] Sparsification of fully-connected layers and separation of convolutional kernels MLP, CNN Georgiev et al [97] Representation sharing MLP Cho and Brand [524] Convolution operation optimization CNN Guo and Potkonjak [525] Filters and classes pruning CNN Li et al [526] Cloud assistance and incremental learning CNN Zen et al [527] Weight quantization LSTM Falcao et al [528] Parallelization and memory sharing Stacked AE Fang et al [529] Model pruning and recovery scheme CNN Xu et al [530] Reusable region lookup and reusable region propagation scheme CNN…”

To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference

“…Numerous approaches have been proposed to accelerate deep inference on embedded devices. These include designing purpose-built hardware to reduce the computation or memory latency [9], compressing a pre-trained model to reduce its storage and memory footprint as well as computational requirements [12], and offloading some, or all, computation to a cloud server [17], [27]. Compared to specialized hardware, model compression techniques have the advantage of being readily deployable on commercial-off-the-self hardware; and compared to computation offloading, compression enables local, on-device inference which in turn reduces the response latency and has fewer privacy concerns.…”