Multi-Modal Dialog State Tracking for Interactive Fashion Recommendation

Abstract

Multi-modal interactive recommendation is a type of task that allows users to receive visual recommendations and express naturallanguage feedback about the recommended items across multiple iterations of interactions. However, such multi-modal dialog sequences (i.e. turns consisting of the system's visual recommendations and the user's natural-language feedback) make it challenging to correctly incorporate the users' preferences across multiple turns. Indeed, the existing formulations of interactive recommender systems suffer from their inability to capture the multi-modal sequential dependencies of textual feedback and visual recommendations because of their use of recurrent neural network-based (i.e., RNNbased) or transformer-based models. To alleviate the multi-modal sequential dependency issue, we propose a novel multi-modal recurrent attention network (MMRAN) model to effectively incorporate the users' preferences over the long visual dialog sequences of the users' natural-language feedback and the system's visual recommendations. Specifically, we leverage a gated recurrent network (GRN) with a feedback gate to separately process the textual and visual representations of natural-language feedback and visual recommendations into hidden states (i.e. representations of the past interactions) for multi-modal sequence combination. In addition, we apply a multi-head attention network (MAN) to refine the hidden states generated by the GRN and to further enhance the model's ability in dynamic state tracking. Following previous work, we conduct extensive experiments on the Fashion IQ Dresses, Shirts, and Tops & Tees datasets to assess the effectiveness of our proposed model by using a vision-language transformer-based user simulator as a surrogate for real human users. Our results show that our proposed MMRAN model can significantly outperform several existing state-of-the-art baseline models. CCS CONCEPTS• Information systems → Recommender systems; • Computing methodologies → Learning from critiques.

Goal-Oriented Multi-Modal Interactive Recommendation with Verbal and Non-Verbal Relevance Feedback

“…Multi-Modal Interactive Recommendation. Recently, multi-modal interactive recommendation has been intensively investigated in the literature, as it can satisfy the users' information needs by effectively eliciting the users' preferences from the visual recommendations (e.g., images of fashion products) and the corresponding verbal and/or non-verbal relevance feedback (e.g., natural-language feedback and likes/dislikes) [7,12,19,31,[45][46][47]53]. These kinds of interactive recommendations are suited for taste-oriented domains such as fashion, where search-type interaction methods are less useful.…”

“…These kinds of interactive recommendations are suited for taste-oriented domains such as fashion, where search-type interaction methods are less useful. Typically, the multi-modal interactive recommendation task focuses on tracking and estimating the users' preferences over time with a state tracker, such as a gated recurrent unit (GRU) [19,45], a long short-term memory (LSTM) [57], a Transformer encoder [43,47], or an RNN-enhanced Transformer [46], in an end-to-end fashion with supervised learning (SL) and/or deep reinforcement learning (DRL) approaches. The representations of visual candidate items and natural-language feedback are initially generated with pretrained models (such as ResNet for image encoding and BERT or GloVe for text encoding), and are then implicitly further tuned along with the recommendation policy optimisation.…”

“…We particularly argue that the existing multi-modal recommendation models [19,46,57] have not effectively addressed the coupling issue of the policy optimisation and representation learning from both the verbal and non-verbal relevance feedback. Such an issue limits these models' ability at incorporating the users' preferences over time.…”

“…State Tracking Stage. To incorporate the users' preferences from the combined text and image representations 𝑐 𝑡 = 𝜓 (𝑎 𝑡,𝑢 , 𝑜 𝑡 ), we leverage a Transformer encoder 𝑇𝑟𝑎𝑛𝐸𝑛𝑐 (•), as in [43,46,47], as a state tracker to track/estimate the interaction states. In particular, the Transformer encoder allows our GOMMIR model to sequentially aggregate the recommendation and feedback information from the multi-modal composition feature 𝑐 𝑡 to attend to the entire feedback history during each interaction turn.…”

“…To summarise, in the GOMMIR model, we maintain the Transformer Encoder for state tracking and the 𝐾𝑁 𝑁𝑠 (•) for sampling as in the state-of-the-art approaches [43,46,47]. Meanwhile, we leverage the CLIP-based multi-modal encoders and a composition network (i.e.…”