The SONICOM Project: Artificial Intelligence-Driven Immersive Audio, From Personalization to Modeling [Applications Corner]

Abstract

Every individual perceives spatial audio differently due in large part to the unique and complex shape of their ears and head. Therefore, high-quality headphone-based spatial audio should be uniquely tailored to each listener in an effective and efficient manner. Artificial Intelligence (AI) is a powerful tool that can be used to drive forward research in spatial audio personalisation. The SONICOM project aims to employ a data-driven approach to link the physiological characteristics of the ear to the individual acoustic filters which allow us to localise sound sources and to perceive them as being located around us. A small amount of data acquired from users could allow personalised audio experiences, and AI could facilitate this by offering a new perspective on the matter. A Bayesian approach to computational neuroscience and binaural sound reproduction will be linked to create a metric for AI-based algorithms that will predict realistic spatial audio quality. Being able to consistently and repeatedly evaluate and quantify the improvements brought by technological advancements, as well as the impact these have on complex interactions in virtual environments, will be key for the development of new techniques and unlocking new approaches to understanding the mechanisms of human spatial hearing and communication.

Exploring the impact of transfer learning on GAN-based HRTF upsampling

“…The SONICOM dataset [5,20] is a publicly-released HRTF dataset which aims at facilitating reproducible research in the spatial acoustics and immersive audio domain by including in a single database HRTFs measured from an increasingly large number of subjects (200 subjects were used in this work).…”

The SONICOM HRTF Dataset

“…Created within the remit of the SONICOM project [19], 1 this technical paper introduces the publicly-released SONI-COM HRTF dataset, which aims at facilitating reproducible research (i.e., allowing researchers to ensure that they can repeat the same analysis multiple times with the same results [20]) in the spatial acoustics and immersive audio domain by including in a single database HRTFs measured from an increasingly large number of subjects (currently 120), as well as headphones transfer functions; 3D scans of ears, head, and torso; and RGB + depth pictures at different angles around the subject's head.…”