Context-aware Image Tweet Modelling and Recommendation

Chen, Tao; He, Xiangnan; Kan, Min-Yen

doi:10.1145/2964284.2964291

Cited by 78 publications

(67 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, multimedia data becomes prevalent on the current Web. For example, products are usually associated with images to attract customers in E-commerce sites [6], and users usually post images or micro-videos to interact with their friends in social media sites [7], [8]. Such multimedia content contains rich visually-relevant signal that can reveal user preference [9], providing opportunities to improve recommender systems that are typically based on collaborative filtering on user behavior data only [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Adversarial Training Towards Robust Multimedia Recommender System

Tang

et al. 2020

IEEE Trans. Knowl. Data Eng.

Self Cite

154

View full text Add to dashboard Cite

With the prevalence of multimedia content on the Web, developing recommender solutions that can effectively leverage the rich signal in multimedia data is in urgent need. Owing to the success of deep neural networks in representation learning, recent advance on multimedia recommendation has largely focused on exploring deep learning methods to improve the recommendation accuracy. To date, however, there has been little effort to investigate the robustness of multimedia representation and its impact on the performance of multimedia recommendation. In this paper, we shed light on the robustness of multimedia recommender system. Using the state-of-the-art recommendation framework and deep image features, we demonstrate that the overall system is not robust, such that a small (but purposeful) perturbation on the input image will severely decrease the recommendation accuracy. This implies the possible weakness of multimedia recommender system in predicting user preference, and more importantly, the potential of improvement by enhancing its robustness. To this end, we propose a novel solution named Adversarial Multimedia Recommendation (AMR), which can lead to a more robust multimedia recommender model by using adversarial learning. The idea is to train the model to defend an adversary, which adds perturbations to the target image with the purpose of decreasing the model's accuracy. We conduct experiments on two representative multimedia recommendation tasks, namely, image recommendation and visually-aware product recommendation. Extensive results verify the positive effect of adversarial learning and demonstrate the effectiveness of our AMR method. Source codes are available in https://github.com/duxy-me/AMR.

show abstract

Section: Introductionmentioning

confidence: 99%

Adversarial Training Towards Robust Multimedia Recommender System

Tang

et al. 2020

IEEE Trans. Knowl. Data Eng.

Self Cite

154

View full text Add to dashboard Cite

show abstract

“…In many image based social networks, images are associated with rich context information, e.g., the text in the image, the hashtags. Researchers proposed to apply factorization machines for image recommendation by considering the rich context information [6]. Recently, deep Convolutional Neural Networks(CNNs) have been successfully applied to analyzing visual imagery by automatic image representation in the modeling process [27].…”

Section: Related Workmentioning

confidence: 99%

A Hierarchical Attention Model for Social Contextual Image Recommendation

Chen

Hong

et al. 2020

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Image based social networks are among the most popular social networking services in recent years. With tremendous images uploaded everyday, understanding users' preferences on user-generated images and making recommendations have become an urgent need. In fact, many hybrid models have been proposed to fuse various kinds of side information (e.g., image visual representation, social network) and user-item historical behavior for enhancing recommendation performance. However, due to the unique characteristics of the user generated images in social image platforms, the previous studies failed to capture the complex aspects that influence users' preferences in a unified framework. Moreover, most of these hybrid models relied on predefined weights in combining different kinds of information, which usually resulted in sub-optimal recommendation performance. To this end, in this paper, we develop a hierarchical attention model for social contextual image recommendation. In addition to basic latent user interest modeling in the popular matrix factorization based recommendation, we identify three key aspects (i.e., upload history, social influence, and owner admiration) that affect each user's latent preferences, where each aspect summarizes a contextual factor from the complex relationships between users and images. After that, we design a hierarchical attention network that naturally mirrors the hierarchical relationship (elements in each aspects level, and the aspect level) of users' latent interests with the identified key aspects. Specifically, by taking embeddings from state-of-the-art deep learning models that are tailored for each kind of data, the hierarchical attention network could learn to attend differently to more or less content. Finally, extensive experimental results on real-world datasets clearly show the superiority of our proposed model. !

show abstract

“…Long-term temporal dynamics (e.g., fashion evolution) and "visual consistently" in session-level user actions are considered in their following works [25,27]. Chen et al [11] utilize visual features for personalized image tweet recommendation. Wang et al [60] explore images for POI recommendation by incorporating visual contents in PMF [47].…”

Section: Visually-aware Recommendationmentioning

confidence: 99%

Mmalfm

Cheng

Chang

Zhu

et al. 2019

ACM Trans. Inf. Syst.

189

View full text Add to dashboard Cite

Personalized rating prediction is an important research problem in recommender systems. Although the latent factor model (e.g., matrix factorization) achieves good accuracy in rating prediction, it suffers from many problems including cold-start, non-transparency, and suboptimal results for individual user-item pairs. In this paper, we exploit textual reviews and item images together with ratings to tackle these limitations. Specifically, we first apply a proposed multi-modal aspect-aware topic model (MATM) on text reviews and item images to model users' preferences and items' features from different aspects, and also estimate the aspect importance of a user towards an item. Then the aspect importance is integrated into a novel aspect-aware latent factor model (ALFM), which learns user's and item's latent factors based on ratings. In particular, ALFM introduces a weight matrix to associate those latent factors with the same set of aspects in MATM, such that the latent factors could be used to estimate aspect ratings. Finally, the overall rating is computed via a linear combination of the aspect ratings, which are weighted by the corresponding aspect importance. To this end, our model could alleviate the data sparsity problem and gain good interpretability for recommendation. Besides, every aspect rating is weighted by its aspect importance, which is dependent on the targeted user's preferences and the targeted item's features. Therefore, it is expected that the proposed method can model a user's preferences on an item more accurately for each user-item pair. Comprehensive experimental studies have been conducted on the Yelp 2017 Challenge dataset and Amazon product datasets. Results show that (1) our method achieves significant improvement compared to strong baseline methods, especially for users with only few ratings; (2) item visual features can improve the prediction performance -the effects of item image features on improving the prediction results depend on the importance of the visual features for the items; and (3) our model can explicitly interpret the predicted results in great detail. 1:3restaurant; while for a cheap restaurant, the expectation on these two aspects would be low. Thus, the user will give higher weights to the aspects of "service" and "ambience" for the expensive restaurant than the cheap one when rating two such restaurants. Therefore, for accurate prediction, it is important to accurately capture the importance of each latent factor for a user towards an item. At first glance, MF achieves the goal as the influence of a factor (e.g., k-th factor) is dependent on both p u,k and q i,k (i.e., p u,k * q i,k ). However, it models the importance of a factor by a fixed value for an item or a user. As a result, it treats each factor of an item with the same importance to all users (i.e., q i,k ); and similarly, each factor of a user is equally important to all items (i.e., p u,k ) in rating prediction. This will lead to sub-optimal results for individual user-item pair.In this work, we attempt t...

show abstract

Context-aware Image Tweet Modelling and Recommendation

Cited by 78 publications

References 29 publications

Adversarial Training Towards Robust Multimedia Recommender System

Adversarial Training Towards Robust Multimedia Recommender System

A Hierarchical Attention Model for Social Contextual Image Recommendation

Mmalfm

Contact Info

Product

Resources

About