Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses

Abstract

Although media content is increasingly produced, distributed, and consumed in multiple combinations of modalities, how individual modalities contribute to the perceived emotion of a media item remains poorly understood. In this paper we present MusicVideos (MuVi), a novel dataset for affective multimedia content analysis to study how the auditory and visual modalities contribute to the perceived emotion of media. The data were collected by presenting music videos to participants in three conditions: music, visual, and audiovisual. Participants annotated the music videos for valence and arousal over time, as well as the overall emotion conveyed. We present detailed descriptive statistics for key measures in the dataset and the results of feature importance analyses for each condition. Finally, we propose a novel transfer learning architecture to train Predictive models Augmented with Isolated modality Ratings (PAIR) and demonstrate the potential of isolated modality ratings for enhancing multimodal emotion recognition. Our results suggest that perceptions of arousal are influenced primarily by auditory information, while perceptions of valence are more subjective and can be influenced by both visual and auditory information. The dataset is made publicly available.

AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges

“…For example, Markis et al have proposed an AI-AMG system with high-level conditional information that is fed into the encoder part of a sequence-to-sequence architecture for generating valence-specific affective music in a more controllable manner (e.g., the music is generated to match a profile of valence values provided by the user) [43]. However, a challenge in this direction is that datasets labelled with reliable affective information are scarce, although some researchers are working on this limitation [10]. Apart from using a conditional architecture, another potential approach to improve controllability is to use the fundamentals of reinforcement learning to train the neural network models [4].…”

MERP: A Music Dataset with Emotion Ratings and Raters’ Profile Information

“…The relationship between music and emotions has been scientifically explored for at least a hundred years (e.g., Seashore [ 7 ]), with a surge of interest beginning in the 1950s by Meyer [ 8 ], and expanding even more widely in recent decades, both in music psychology [ 9 , 10 , 11 ] and Music Information Retrieval (MIR) [ 5 , 12 , 13 , 14 ]. Generally, there are two main ways of capturing and representing emotion in music: categorical and dimensional [ 15 ].…”

“…More recently, [ 25 ] provided a method for mapping discrete emotional terms onto Russell’s dimensional model. Finally, Chua et al [ 5 ]’s MuVi dataset provides both dynamic ratings along the valence and arousal dimensions throughout the song, as well as static emotion labels (categorical labels for the entire song) for music and video stimuli.…”

“…Despite the surge of interest in MER, the number of datasets with emotion annotations is limited, especially those with dynamic annotations, i.e., ratings that are collected continuously throughout the piece. To the authors’ knowledge, the DEAM dataset (1802 fragments by 15–32 annotators) [ 3 ], the Moodswings dataset (240 fragments by 7–23 annotators) [ 4 ], and the MuVi dataset (81 fragments by 5–9 annotators) [ 5 ] are the only datasets that offer music with dynamic emotion (arousal and valence) annotations. For a more complete overview of existing music datasets with emotion annotations, the reader is referred to Chua et al [ 5 ].…”

“…To the authors’ knowledge, the DEAM dataset (1802 fragments by 15–32 annotators) [ 3 ], the Moodswings dataset (240 fragments by 7–23 annotators) [ 4 ], and the MuVi dataset (81 fragments by 5–9 annotators) [ 5 ] are the only datasets that offer music with dynamic emotion (arousal and valence) annotations. For a more complete overview of existing music datasets with emotion annotations, the reader is referred to Chua et al [ 5 ]. In order to train any powerful machine learning model, the availability of big datasets is crutial.…”