Multi-modal similarity fusion for user behavior sequence modeling

Abstract

With the pervasive use of the Internet, recommendation systems have gained increasing importance in people's daily lives. Among the crucial tasks in recommendation systems, click-through rate prediction stands out as it directly influences their effectiveness. Recent studies have revealed that incorporating user behavior sequences can substantially enhance the accuracy of click-through rate prediction models. However, existing models overlook user preferences for textual and visual information, which hampers the acquisition of a comprehensive representation of user interests. Consequently, this limitation results in suboptimal model accuracy. In this paper, we propose a unified framework for modeling multi-modal user behavior sequences. Our framework leverages a unified cross-modal pre-trained model for feature extraction and employs a multi-modal similarity-enhanced attention mechanism to capture users' preferences across various modalities. We conduct extensive experiments on large-scale real-world datasets to validate the effectiveness of our approach. Compared to other state-of-the-art click-through rate estimation algorithms, our model achieves an approximate 1.22% improvement in AUC, thereby significantly enhancing the accuracy of the recommendation model.