Real time spectrogram inversion on mobile phone

Abstract

With the growth of computing power on mobile phones and privacy concerns over user's data, on-device real time speech processing has become an important research topic. In this paper, we focus on methods for real time spectrogram inversion, where an algorithm receives a portion of the input signal (e.g., one frame) and processes it incrementally, i.e., operating in streaming mode. We present a real time Griffin Lim(GL) algorithm using a sliding window approach in STFT domain. The proposed algorithm is 2.4x faster than real time on the ARM CPU of a Pixel4. In addition we explore a neural vocoder operating in streaming mode and demonstrate the impact of looking ahead on perceptual quality. As little as one hop size (12.5ms) of lookahead is able to significantly improve perceptual quality in comparison to a causal model. We compare GL with the neural vocoder and show different trade-offs in terms of perceptual quality, on-device latency, algorithmic delay, memory footprint and noise sensitivity. For fair quality assessment of the GL approach, we use input log magnitude spectrogram without mel transformation. We evaluate presented real time spectrogram inversion approaches on clean, noisy and atypical speech.

Learning Temporal Resolution in Spectrogram for Audio Classification

“…Besides, the potential of the high-resolution spectrogram, e.g., with a one-millisecond hop size, is still unclear. Some popular choices of hop size including 10 ms (Böck et al 2012;Kong et al 2020;Gong, Chung, and Glass 2021a) and 12.5 ms (Rybakov et al 2022). Previous studies (Kong et al 2020;Ferraro et al 2021) show classification performance can be steadily improved with the increase of resolution.…”