Edge emotion recognition: applying fast Fourier transform on speech Mel spectrograms to classify emotion on a Raspberry Pi for near real-time analytics

Abstract

Many people and machines are inherently unable to interpret socio-affective cues such as tone of voice. Thoughtful adoption of intelligent technologies may improve the conversation. Since direct communication often occurs via edge devices, where an additional network connection is often not guaranteed, we now describe a real-time processing method that captures and evaluates emotions in a speech via a terminal device such as the Raspberry Pi computer. In this article, we also present the current state of research on speech emotional recognition. We examine audio files from five important emotional speech databases and visualize them in situ with dB-scaled Mel spectrograms using TensorFlow and Matplotlib. Audio files are transformed using the fast Fourier transform method to generate spectrograms. For classification, a support vector machine kernel and a CNN with transfer learning are selected. The accuracy of this classification is 70% and 77%, respectively, a good value related to the execution of the algorithms on an edge device instead of on a server. On a Raspberry Pi, it took less than one second to evaluate pure emotion in speech using machine learning and the corresponding visualization, suggesting the speaker's emotional state.