Music/speech classification using high-level features derived from fmri brain imaging

Abstract

With the availability of large amount of audio tracks through a variety of sources and distribution channels, automatic music/speech classification becomes an indispensable tool in social audio websites and online audio communities. However, the accuracy of current acoustic-based low-level feature classification methods is still rather far from satisfaction. The discrepancy between the limited descriptive power of low-level features and the richness of high-level semantics perceived by the human brain has become the 'bottleneck' problem in audio signal analysis. In this paper, functional magnetic resonance imaging (fMRI) which monitors the human brain's response under the natural stimulus of music/speech listening is used as high-level features in the brain imaging space (BIS). We developed a computational framework to model the relationships between BIS features and low-level features in the training dataset with fMRI scans, predict BIS features of testing dataset without fMRI scans, and use the predicted BIS features for music/speech classification in the application stage. Experimental results demonstrated the significantly improved performance of music/speech classification via predicted BIS features than that via the original low-level features.

Melodic Contour and Mid-Level Global Features Applied to the Analysis of Flamenco Cantes

“…The authors recognized the need to introduce mid-level features in automatic music classification and thus design appropriates distances. Xi et al (2012) also addressed the problem where high-level features were derived from fMRI brain imaging.…”

Merging Neuroimaging and Multimedia: Methods, Opportunities, and Challenges

“…The authors reported that sound category information that is not detectable with conventional contrast-based analysis methods could be detected with multivoxel pattern analyses. In [49], the authors used fMRI to monitor the brain's response to natural music and speech listening and developed a computational framework to model the relationships between neuroimaging features and low-level acoustic features in the training dataset with fMRI scans. Then, they predicted neuroimaging features of a testing dataset without fMRI scans, and used the predicted neuroimaging features for music/speech classification.…”

“…Existing systems that merged neuroimaging and multimedia, e.g., [26], [27], [49], assumed temporal stationarity of the brain's response for each multimedia clip, e.g., functional connectivities or other measurements were computed over the entire scan and used to characterize the brain's responses over the whole multimedia segment. However, [82] and [83] have shown that functional brain connectivities are under dynamic changes in different time scales.…”

“…Experimental results have demonstrated that fMRI-derived semantic features were able to narrow the semantic gaps. It has also been demonstrated that the GPR can effectively map the intrinsic relationships between acoustic features extracted in music/speech streams and fMRIderived features [49].…”

Decoding Auditory Saliency from FMRI Brain Imaging

“…Moreover, there have been efforts in studying the neural basis of audio perception and understanding [10,11]. As reported in [10,11], music perception and understanding involve widely distributed human brain areas and functional interactions among large scale brain networks, which provide the theoretical foundation for deriving whole brain semantic features to perform low-level feature decoding.…”