Counterfactually Evaluating Explanations in Recommender Systems

Abstract

Modern recommender systems face an increasing need to explain their recommendations. Despite considerable progress in this area, evaluating the quality of explanations remains a significant challenge for researchers and practitioners. Prior work mainly conducts human study to evaluate explanation quality, which is usually expensive, time-consuming, and prone to human bias. In this paper, we propose an offline evaluation method that can be computed without human involvement. To evaluate an explanation, our method quantifies its counterfactual impact on the recommendation. To validate the effectiveness of our method, we carry out an online user study. We show that, compared to conventional methods, our method can produce evaluation scores more correlated with the real human judgments, and therefore can serve as a better proxy for human evaluation. In addition, we show that explanations with high evaluation scores are considered better by humans. Our findings highlight the promising direction of using the counterfactual approach as one possible way to evaluate recommendation explanations.

Forgetting Fast in Recommender Systems

“…Here the term counterfactual means "Had you not interacted with those items, you would not receive this recommendation. " It has been found by a recent user study that counterfactual recommendation explanations are better correlated with real human judgments [22].…”

Sentiment analysis and opinion mining on educational data: A survey

“…Another area where MU may be important for RS is in the development of more transparent and fair models [217]. MU can help to improve the interpretability and explainability of RS [218]. Unlearning can help to make these models more transparent by allowing them to remove irrelevant or misleading features and focus on the most important factors for making accurate recommendations [219].…”