Simple Pooling Front-Ends for Efficient Audio Classification

Abstract

Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios. Most existing approaches are designed to reduce the size of audio neural networks using methods such as model pruning. In this work, we show that instead of reducing model size using complex methods, eliminating the temporal redundancy in the input audio features (e.g., mel-spectrogram) could be an effective approach for efficient audio classification. To do so, we proposed a family of simple pooling front-ends (SimPFs) which use simple non-parametric pooling operations to reduce the redundant information within the mel-spectrogram. We perform extensive experiments on four audio classification tasks to evaluate the performance of SimPFs. Experimental results show that SimPFs can achieve a reduction in more than half of the number of floating point operations (FLOPs) for off-the-shelf audio neural networks, with negligible degradation or even some improvements in audio classification performance.

Learning Temporal Resolution in Spectrogram for Audio Classification

“…Compression of mel-spectrogram temporal dimension can lead to a considerable speed up on training and inference (Liu et al 2023), which has significant promise in on-device scenarios. In this section, we evaluate the effectiveness of DiffRes in compressing temporal dimensions and classification performance.…”

“…There are plenty of studies on learning a suitable frequency resolution with a similar spirit (Stevens, Volkmann, and Newman 1937;Sainath et al 2013;Ravanelli and Bengio 2018b;Zeghidour et al 2021). Most previous studies focus on investigating the effect of different temporal resolutions (Kekre et al 2012;Huzaifah 2017;Ilyashenko et al 2019;Liu et al 2023). Huzaifah (2017) observe the optimal temporal resolution for audio classification is class dependent.…”

“…Kazakos et al (2021) propose a two-stream architecture that processes both fine-grained and coarse-resolution spectrogram and shows the state-of-the-art result on VGG-Sound (Chen et al 2020). Recently, Liu et al (2023) proposed a nonparametric spectrogram-pooling-based module that can improve classification efficiency with negligible performance degradation. However, these approaches are generally built on a fixed temporal resolution, which is not always optimal for diverse sounds in the world.…”