Photosense: Make sense of your photos with enriched harmonic music via emotion association

Abstract

This paper proposes a novel audiovisual presentation system, called PhotoSense, to enrich photo navigation experience by associating emotionally harmonic music with a given photo collection. Different from many conventional photo visualization systems which predominantly focus on the visual elements for presentation, we explore both visual and aural perspectives which can enhance the browsing experience from each other. This is achieved by building an emotion space shared by visual and aural domains, and a set of emotion classifiers which can associate each visual and aural element with this space. Furthermore, we design a sequence matching algorithm to associate a set of music with a photo collection by maximizing similarity in the emotion space. PhotoSense represents one of the first mash-up applications which build a natural connection between the ever increasing personal photo collections on the Web and music-sharing sites. Experiments show that PhotoSense provides better browsing experience for photo collections.

What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay?

“…The emotion detected through the stroke behavior 31:24 Y. Gao et al could be used to tag the stored material and to identify clusters of stored information (music and photos) that reflect similar emotions [Su et al 2011]. Such metatagging could also be used to facilitate crowdsourcing of media information and to build better recommendation systems [Brew et al 2010].…”

Exploring the Opportunities of AR for Enriching Storytelling with Family Photos between Grandparents and Grandchildren

“…(4) Sound effects (ST): Previous literature [12,71] found that playing background sounds related to photos could enhance the atmosphere of the story and make the listener and speaker feel immersed in the scene, thereby enhancing the effect of communication. Different types of background sounds for different photos were provided in the probe (e.g., old songs of the same age as the photo of a park, the sound of waves and seagulls for the photo on the beach, and the cries of hawkers for a market photo).…”

Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

“…They separately extracted hand-crafted features, learned emotion classifiers, and composited images and music based on the predicted emotions. Many methods follow this pipeline [34,48,53,61,86]. They (1) extracted more discriminative emotion features, such as low-level color [9,34,48,53,61] and mid-level principlesof-art [86] for image; (2) employed different emotion representation models, from categorical states [9,34,53,61] to dimensional space [48,86]; (3) correspondingly learned different classifiers, from Support Vector Machine [9], Naive Bayes, and Decision Tree [53] to Support Vector Regression [86]; and (4) used different composition strategies to match image and music, from emotion category comparison [9,34,53,61] to Euclidean distance [48,86].…”

“…Such emotion-based matching is essential for various applications [57], such as affective cross-modal retrieval, emotion-based multimedia slideshow, and emotion-aware recommendation systems. The early emotion-based matching methods mainly employ a shallow pipeline [9,34,48,53,61,86], i.e. extracting hand-crafted features and training matching classifiers (or training emotion classifiers for both modalities and then learning matching similarities).…”