Saliency-maximized audio visualization and efficient audio-visual browsing for faster-than-real-time human acoustic event detection

Abstract

Examining large audio archives is a challenging task for humans owing to the limitations of human audition. We explore an innovative approach to engage both human vision and audition for audio browsing, which significantly improves human acoustic event detection in long audio recordings. In particular we visualize the data as a saliency-maximized spectrogram, accessed at different temporal scales using a special audio browser that also allows rapid zooming across scales from hours to milliseconds.The saliency-maximized audio spectrogram lets humans quickly search for and detect events in audio recordings. By rendering target events as visually salient patterns, this representation minimizes the time and effort needed to visually examine a recording. This transformation maximizes the mutual information between the spectrogram of an isolated target event and the estimated saliency of the overall visual representation.When subjects are shown spectrograms that are saliency-maximized instead of the original spectrograms, they perform significantly better in a 1/10-real-time acoustic event detection task.

Motion trails from time-lapse video

“…This paper presents a way to greatly reduce such risk: video processing that increases the saliency of transient events, anomalies, and patterns, letting them be seen at a glance during fast-forward or even in still frames, instead of taxing short-term memory. The approach is inspired by the solution to a similar problem: in long audio recordings, increasing the saliency of brief anomalous sounds [10].…”

Artificial Synaesthesia: An exploration of machine learning image synthesis for soundscape audio visualisation

“…Similarly, altering spectrograms to make visual audio easier and quicker to scan for audio events has been explored (Lin et al, 2013). Phillips et al (2018) use visualisations of data from long-duration audio allowing for faster content identification and organisation when monitoring audio change.…”

“…Identification of sound events in long duration field recordings and their visualisation have been explored in the past, as long-duration recordings have become more accessible (Phillips et al, 2018;Towsey et al, 2014) but Marsland et al (2019) applied audio classification to the process to further improve the facilitation of analysis. As well as classification, machine learning also tends to be utilised for smaller computational tasks like noise reduction (Lin et al, 2013) and audio feature extraction (Dhiraj et al, 2019), but never as the main visualisation tool.…”