Exploring the Semantic Gap for Movie Recommendations

Abstract

In the last years, there has been much a ention given to the semantic gap problem in multimedia retrieval systems. Much e ort has been devoted to bridge this gap by building tools for the extraction of high-level, semantics-based features from multimedia content, as low-level features are not considered useful because they deal primarily with representing the perceived content rather than the semantics of it.In this paper, we explore a di erent point of view by leveraging the gap between low-level and high-level features. We experiment with a recent approach for movie recommendation that extract lowlevel Mise-en-Scène features from multimedia content and combine it with high-level features provided by the wisdom of the crowd.To this end, we rst performed an o ine performance assessment by implementing a pure content-based recommender system with three di erent versions of the same algorithm, respectively based on (i) conventional movie a ributes, (ii) mise-en-scène features, and (iii) a hybrid method that interleaves recommendations based on movie a ributes and mise-en-scène features. In a second study, we designed an empirical study involving 100 subjects and collected data regarding the quality perceived by the users. Results from both studies show that the introduction of mise-en-scène features in conjunction with traditional movie a ributes improves both o ine and online quality of recommendations.

Movie genome: alleviating new item cold start in movie recommendation

“…Note that in this study, the tags feature is considered because, as stated, the study's focus is no longer on new movie recommendation (as in study A) and tags serve as a rich semantic baseline. Our preliminary studies in a similar direction have been published in Elahi et al (2017) which focused on a single visual modality (Elahi et al 2017), and in Deldjoo et al (2018b), which used a lower number of participants (74 vs. 101). In addition, compared to Deldjoo et al (2018b), we performed better sanity checks and removed unreliable user input.…”

“…In Deldjoo et al (2015aDeldjoo et al ( , b, 2016a, Elahi et al (2017) and Cremonesi et al (2018), we proposed a CB-MRS that implements a movie filter according to average shot length (measure of camera motion), color variation, lighting key (measure of contrast), and motion (measure of object and camera motion). The proposed features were originally used in the field of multimedia retrieval for movie genre classification (Rasheed et al 2005) and have a stylistic nature which is believed to be in accordance with applied media aesthetics (Zettl 2013) for conveying communication effects and simulating different feelings in the viewers.…”

“…In a different research line, in Elahi et al (2017), we designed an online movie recommender system which incorporates mise-en-scène visual features for the evaluation of recommendations by real users. We performed an offline performance assessment by implementing a pure CB-MRS with three different versions of the same algorithm, respectively based on (i) conventional movie attributes, (ii) mise-en-scène visual features, and (iii) a hybrid method that interleaves recommendations based on the previously noted features.…”

“…Results from both studies showed that the introduction of mise-en-scène, together with traditional movie attributes, improves the quality of both offline and online recommendations. However, the main limitation of Elahi et al (2017) is that we used basic late fusion by interleaving the recommendations to combine recommendations generated by different CBF systems.…”

Current challenges and visions in music recommender systems research

“…7 Regardless of whether the feature extraction process is performed automatically or manually, this approach is advantageous not only to address the new item problem but also because an accurate feature representation can be highly predicative of users' tastes and interests which can be leveraged in the subsequent information ltering stage [6]. An advantage of music to video is that features in music is limited to a single audio channel, compared to audio and visual channels for videos adding a level complexity to the content analysis of videos explored individually or multimodal in di erent research works [47,48,60,128].…”