Enhancing Video Recommendation Using Multimedia Content

Abstract

Video recordings are complex media types. When we watch a movie, we can effortlessly register a lot of details conveyed to us (by the author) through different multimedia channels, in particular, the audio and visual modalities. To date, majority of movie recommender systems use collaborative filtering (CF) models or content-based filtering (CBF) relying on metadata (e.g., editorial such as genre or wisdom of the crowd such as user-generated tags) at their core since they are human-generated and are assumed to cover the 'content semantics' of movies by a great degree. The information obtained from multimedia content and learning from muli-modal sources (e.g., audio, visual and metadata) on the other hand, offers the possibility of uncovering relationships between modalities and obtaining an in-depth understanding of natural phenomena occurring in a video. These discerning characteristics of heterogeneous feature sets meet users' differing information needs. In the context of this Ph.D. thesis [9], which is briefly summarized in the current extended abstract, approaches to automated extraction of multimedia information from videos and their integration with video recommender systems have been elaborated, implemented, and analyzed. Variety of tasks related to movie recommendation using multimedia content have been studied. The results of this thesis can motivate the fact that recommender system research can benefit from knowledge in multimedia signal processing and machine learning established over the last decades for solving various recommendation tasks.

An overview of video recommender systems: state-of-the-art and research issues

“…Videos consist of three modalities: (1) Aural (audio information), (2) Visual (visual frames), and (3) Textual (textual descriptions and metadata), which can be expressed in varying degrees of semantic detail. This characteristic makes videos multi-modal , as they include all three modalities, whereas a music piece without lyrics is uni-modal , as it only features aural elements (Deldjoo, 2020 ).…”

“…Based on the classification in Deldjoo ( 2020 ), video features can be categorized into groups based on their modality and semantic expressiveness: (1) Low-level features describe the raw signal of a video, representing its stylistic properties. (2) Mid-level features require interpretation knowledge and are derived from low-level features, representing syntactic features.…”

Interest Level Estimation via Multi-Modal Gaussian Process Latent Variable Factorization

“…The comparative methods other than GPLVMF only perform construction of the latent space, that is, interest level estimation and the latent space construction were separately conducted. In fact, since previous approaches such as [14][15][16] separately perform CCA-based latent space construction and interest level estimation, comparative methods other than GPLVMF were applied the tensor completion which is often used in the field of recommendation systems [28][29][30] to estimate interest levels. Note that the parameters Qv, Qw, Q c v , Q c w , Q b v and Q b w were set to 5. a1, a2 and a3 were set to 0.5, 0.3 and 0.1, respectively, and others were set to 0.…”

“…Furthermore, mGPLVF incorporates a factorization machine framework [13], which is widely used for estimating interest levels and user preferences, into the latent space construction. Therefore, since the optimal estimator for the latent space can be constructed, mGPLVF leads to more accurate estimation than previously proposed approaches [14][15][16] which separately perform CCA-based latent space construction [8] and estimation of interest levels. To the best of our knowledge, the proposed method is the first approach to simultaneously perform both the latent space construction and interest level estimation using contents and behav- ior information.…”

Using movie trailers to enhance both f2f and online language learning outcomes in an EFL speaking class

“…One of such opportunities is movie trailers. A movie trailer differs from other multimedia contents (e.g., movie) and usually contains the most exciting, funny, or attention-grabbing scenes, which is a strong argument against the representativeness of the trailers for the entire movie (Deldjoo, 2020). Based on the advantages of using movie trailers, it is worth investigating their effects on language learning outcomes both in f2f and online learning.…”