Music thumbnailing via neural attention modeling of music emotion

Huang, Yu-Siang; Chou, Szu-Yu; Yang, Yi-Hsuan

doi:10.1109/apsipa.2017.8282049

Cited by 16 publications

(13 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The following observations can be made by comparing the results within the second group in Table 2: 1) LF works better than EF, 2) the positional encoding is useful for NAM, 3) with the positional encoding, NAM slightly outperforms RNAM. This validates our previous choice of LF over EF (Huang et al, 2017b), and shows that the attention mechanism can be simplified by using fully-connected layers.…”

Section: Resultssupporting

confidence: 86%

“…We took a different approach in a previous work (Huang et al, 2017b) by utilizing the possible connections between emotions and song highlights. The idea is to firstly use a data set with emotion labels to train a neural network model for music emotion classification (Yang and Liu, 2013).…”

Section: Related Workmentioning

confidence: 99%

“…Their attention mechanism assessed how each audio chunk contributes Figure 1: Architecture of two attention-based models using different fusion methods for highlight extraction. We note that model (a) was used by Huang et al (2017b) and model (b) by Ha et al (2017).…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Pop Music Highlighter: Marking the Emotion Keypoints

Huang

Chou

Yang

2018

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

The goal of music highlight extraction, or thumbnailing, is to extract a short consecutive segment of a piece of music that is somehow representative of the whole piece. In a previous work, we introduced an attention-based convolutional recurrent neural network that uses music emotion classification as a surrogate task for music highlight extraction, assuming that the most emotional part of a song usually corresponds to the highlight. This paper extends our previous work in the following two aspects. First, methodology-wise we experiment with a new architecture that does not need any recurrent layers, making the training process faster. Moreover, we compare a late-fusion variant and an early-fusion variant to study which one better exploits the attention mechanism. Second, we conduct and report an extensive set of experiments comparing the proposed attention-based methods to a heuristic energy-based method, a structural repetition-based method, and three other simple feature-based methods, respectively. Due to the lack of public-domain labeled data for highlight extraction, following our previous work we use the RWC-Pop 100-song data set to evaluate how the detected highlights overlap with any chorus sections of the songs. The experiments demonstrate superior effectiveness of our methods over the competing methods. For reproducibility, we share the code and the pre-trained model at https://github. com/remyhuang/pop-music-highlighter/.

show abstract

Section: Resultssupporting

confidence: 86%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Pop Music Highlighter: Marking the Emotion Keypoints

Huang

Chou

Yang

2018

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the case of music, the author considers music enjoyment as a prediction problem based on the music that the listener has consumed before, with the hypothesis that music enjoyment consists of predicting the sounds that come next and verifying whether these expectations are met (Meyer, 1957). Huang et al (2017) use emotions to find song fragments that represent the whole song. With CNNs, the authors use emotion recognition as a way to compare if a given part of a song corresponds to the chorus section.…”

Section: Working With Human Subjectivitymentioning

confidence: 99%

Machine learning for music genre: multifaceted review and experimentation with audioset

Ramírez

Flores

2019

J Intell Inf Syst

View full text Add to dashboard Cite

Music genre classification is one of the sub-disciplines of music information retrieval (MIR) with growing popularity among researchers, mainly due to the already open challenges. Although research has been prolific in terms of number of published works, the topic still suffers from a problem in its foundations: there is no clear and formal definition of what genre is. Music categorizations are vague and unclear, suffering from human subjectivity and lack of agreement. In its first part, this paper offers a survey trying to cover the many different aspects of the matter. Its main goal is give the reader an overview of the history and the current state-of-the-art, exploring techniques and datasets used to the date, as well as identifying current challenges, such as this ambiguity of genre definitions or the introduction of human-centric approaches. The paper pays special attention to new trends in machine learning applied to the music annotation problem. Finally, we also include a music genre classification experiment that compares different machine learning models using Audioset.

show abstract

“…We regard this architecture as an emotion learning model [10], which is trained over the MER31K dataset, using emotion tags from AllMusic 1 . The detail of selecting audio segments achieved by emotion learning model is shown in Fig.3.…”

Section: A Neural Attention Modelingmentioning

confidence: 99%

Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA

Zeng

Oyama

2018

2018 IEEE International Symposium on Multimedia (ISM)

View full text Add to dashboard Cite

Deep learning has successfully shown excellent performance in learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities, such as audio and video, should be taken into account. Music video retrieval by a given musical audio is a natural way to search and interact with music contents. In this work, we study cross-modal music video retrieval in terms of emotion similarity. Particularly, an audio of an arbitrary length is used to retrieve a longer or full-length music video. To this end, we propose a novel audio-visual embedding algorithm by Supervised Deep Canonical Correlation Analysis (S-DCCA) that projects audio and video into a shared space to bridge the semantic gap between audio and video. This also preserves the similarity among audio and visual contents from different videos with the same class label and the temporal structure. The contribution of our approach is mainly manifested in the two aspects: i) We propose to select top k audio chunks by attention-based Long Short-Term Memory (LSTM) model, which can represent good audio summarization with local properties. ii) We propose an end-to-end deep model for crossmodal audio-visual learning where S-DCCA is trained to learn the semantic correlation between audio and visual modalities. Due to the lack of music video dataset, we construct 10K music video dataset from YouTube 8M dataset. Some promising results such as MAP and precision-recall show that our proposed model can be applied to music video retrieval.

show abstract

Music thumbnailing via neural attention modeling of music emotion

Cited by 16 publications

References 7 publications

Pop Music Highlighter: Marking the Emotion Keypoints

Pop Music Highlighter: Marking the Emotion Keypoints

Machine learning for music genre: multifaceted review and experimentation with audioset

Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA

Contact Info

Product

Resources

About