Self-Supervised Learning for Multimedia Recommendation

Tao, Zhulin; Liu, Xiaohao; Yewei, Xia,; Wang, Xiang; Yang, Lifang; Huang, Xianglin; Chua, Tat-Seng

doi:10.1109/tmm.2022.3187556

Cited by 57 publications

(24 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• MMGCL [49]: It incorporates the graph contrastive learning into recommender through modality edge dropout and masking. • SLMRec [33]: This method designs data augmentation on multimodal content with two components, i.e., noise perturbation over features and multi-modal pattern uncovering augmentation.…”

Section: Evaluation Protocolsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Self-Supervised Learning for Recommendation

Wei,

Huang,

Xia

et al. 2023

Preprint

View full text Add to dashboard Cite

The online emergence of multi-modal sharing platforms (e.g., Tik-Tok, Youtube) is powering personalized recommender systems to incorporate various modalities (e.g., visual, textual and acoustic) into the latent user representations. While existing works on multimodal recommendation exploit multimedia content features in enhancing item embeddings, their model representation capability is limited by heavy label reliance and weak robustness on sparse user behavior data. Inspired by the recent progress of self-supervised learning in alleviating label scarcity issue, we explore deriving selfsupervision signals with effectively learning of modality-aware user preference and cross-modal dependencies. To this end, we propose a new Multi-Modal Self-Supervised Learning (MMSSL) method which tackles two key challenges. Specifically, to characterize the inter-dependency between the user-item collaborative view and item multi-modal semantic view, we design a modality-aware interactive structure learning paradigm via adversarial perturbations for data augmentation. In addition, to capture the effects that user's modality-aware interaction pattern would interweave with each other, a cross-modal contrastive learning approach is introduced to jointly preserve the inter-modal semantic commonality and user preference diversity. Experiments on real-world datasets verify the superiority of our method in offering great potential for multimedia recommendation over various state-of-the-art baselines. The implementation is released at: https://github.com/HKUDS/MMSSL.

show abstract

Section: Evaluation Protocolsmentioning

confidence: 99%

“…Amazon-Baby Allrecipes Tiktok Metrics R@20 N@20 R@20 N@20 R@20 N@20 SLMRec-w-ASL 0.0835 0.0351 0.0342 0.0125 0.0871 0.0366 SLMRec-w-CL 0.0790 0.0331 0.0327 0.0119 0.0853 0.0359 SLMRec-w-CL directly uses the output of GNNs, without using the spatial transformation in Eq.10 (SLMRec) [33]. SLMRec-ASL adds operations in Eq.2,3,4,5,7 of MMSSL.…”

Section: Datamentioning

confidence: 99%

Multi-Modal Self-Supervised Learning for Recommendation

Wei,

Huang,

Xia

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Amazon VBPR [15], VMCF [38], AMR [47], VECF [7], PAMD [14], PMGT [32], LATTICE [67], JRL [69], BM3 [80], GraphCAR [62], DVBPR [19], MAML [27], DMRL [26], MVGAE [64], MMCPR [33], HCGCN [36], FREEDOM [73], DRAGON [71] Kwai MMGCN [59], SLMRec [48], GRCN [58], EliMRec [31], InvRL [10], A2BM2GL [3] Tiktok MMGCN [59], DualGNN [53], MGAT [49], SLMRec [48], GRCN [58], MMGCL [65], EgoGCN [5], EliMRec [31], InvRL [10], A2BM2GL [3], LUDP [24] MovieLens MMGCN [59], DualGNN [53], MGAT [49], SLMRec [48], GRCN [58], MMGCL [65], MKGAT [46], EgoGCN [5], EliMRec [31], InvRL [10], A2BM2GL [3], LUDP [24] Yelp PAMD [...…”

Section: Dataset Modelsmentioning

confidence: 99%

“…Different from the traditional recommendation, those applications have utilized the item multimodal content information like the video frames, audio track and item descriptions. MMGCN [59], MGAT [49], DualGNN [53] and SLMRec [48] are the micro video recommendation models that utilize description, captions, audio, and frames inside the video to model the multimodal user preference on the micro-video. Fashion recommendation faces difficulties to build an efficient recommender cause of the complex features involved and the subjectivity.…”

Section: Introductionmentioning

confidence: 99%

A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions

Zhou¹,

Zhou²,

Zeng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recommendation systems have become popular and effective tools to help users discover their interesting items by modeling the user preference and item property based on implicit interactions (e.g., purchasing and clicking). Humans perceive the world by processing the modality signals (e.g., audio, text and image), which inspired researchers to build a recommender system that can understand and interpret data from different modalities. Those models could capture the hidden relations between different modalities and possibly recover the complementary information which can not be captured by a uni-modal approach and implicit interactions. The goal of this survey is to provide a comprehensive review of the recent research efforts on the multimodal recommendation. Specifically, it shows a clear pipeline with commonly used techniques in each step and classifies the models by the methods used. Additionally, a code framework has been designed that helps researchers new in this area to understand the principles and techniques, and easily runs the SOTA models. Our framework is located at: https://github.com/enoche/MMRec.

show abstract

“…In embedding-based approaches, additional user and item features are used to inform either the neural network producing these embeddings or the neural network that performs recommendations based on them; such approaches include the widely used DeepCoNN [83], YouTube recommendations [10], embeddings based on topic models [72], and more. As the primary baseline for this work, we selected the Personalized Content Discovery (PCD) model [29] because it was tailored to a similar problem of content discovery for brands but also note several other works that extend recommender systems with extra features and extra data modalities [8,20,28,34,45,46,68,69,73,77,81].…”

Section: Related Workmentioning

confidence: 99%

Personality-Driven Social Multimedia Content Recommendation

Yang,

Nikolenko,

Huang

et al. 2022

Preprint

View full text Add to dashboard Cite

Social media marketing plays a vital role in promoting brand and product values to wide audiences. In order to boost their advertising revenues, global media buying platforms such as Facebook Ads constantly reduce the reach of branded organic posts, pushing brands to spend more on paid media ads. In order to run organic and paid social media marketing efficiently, it is necessary to understand the audience, tailoring the content to fit their interests and online behaviours, which is impossible to do manually at a large scale. At the same time, various personality type categorization schemes such as the Myers-Briggs Personality Type indicator make it possible to reveal the dependencies between personality traits and user content preferences on a wider scale by categorizing audience behaviours in a unified and structured manner. Still, McKinsey-style manual categorization is a very labour-intensive task that is probably impractical in a real-world scenario, so automated incorporation of audience behaviour and personality mining into industrial applications is necessary. This problem is yet to be studied in depth by the research community, while the level of impact of different personality traits on content recommendation accuracy has not been widely utilised and comprehensively evaluated so far. Even worse, there is no dataset available for the research community to serve as a benchmark and drive further research in this direction. The present study is one of the first attempts to bridge this important industrial gap, contributing not just a novel personality-driven content recommendation approach and dataset, but also facilitating a real-world ready solution which is scalable and sufficiently accurate to be applied in real-world settings. Specifically, in this work we investigate the impact of human personality traits on the content recommendation model by applying a novel personality-driven multi-view content recommender system called Personality Content Marketing Recommender Engine, or PersiC. Our experimental results and real-world case study demonstrate not just PersiC's ability to perform efficient human personality-driven multi-view content recommendation, but also allow for actionable

show abstract

Self-Supervised Learning for Multimedia Recommendation

Cited by 57 publications

References 31 publications

Multi-Modal Self-Supervised Learning for Recommendation

Multi-Modal Self-Supervised Learning for Recommendation

A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions

Personality-Driven Social Multimedia Content Recommendation

Contact Info

Product

Resources

About