TTRS: Tinkoff Transactions Recommender System benchmark

Abstract

Over the past decade, tremendous progress has been made in inventing new RecSys methods. However, one of the fundamental problems of the RecSys research community remains the lack of applied datasets and benchmarks with well-defined evaluation rules and metrics to test these novel approaches. In this article, we present the TTRS -Tinkoff Transactions Recommender System benchmark. This financial transaction benchmark contains over 2 million interactions between almost 10,000 users and more than 1,000 merchant brands over 14 months. To the best of our knowledge, this is the first publicly available financial transactions dataset. To make it more suitable for possible applications, we provide a complete description of the data collection pipeline, its preprocessing, and the resulting dataset statistics. We also present a comprehensive comparison of the current popular RecSys methods on the next-period recommendation task and conduct a detailed analysis of their performance against various metrics and recommendation goals. Last but not least, we also introduce Personalized Item-Frequenciesbased Model (Re)Ranker -PIFMR, a simple yet powerful approach that has proven to be the most effective for the benchmarked tasks.

CVTT: Cross-Validation Through Time

“…Meanwhile, many recent works [20,50,52] only focus on temporal user evaluation, which may be affected by data leak issue [40]. Finally, available global temporal-based benchmarks [28] lack time-dependent evaluation over time, reducing the final performance to a single-value metric.…”

“…Finally, if we are curious about user interests over time, we could group user interactions into time-based baskets and predict their interactions for the next predefined period. This is the next-period recommendation task [28,66]. In this work, we focus on the latter task, as it is capable of natively integrating with the global temporal split and is actively used in relevant works [23,52].…”

“…• Ta- This dataset is widely used for next-basket prediction research [20,21,30]. • TTRS -recently proposed financial transactions dataset [28]. It contains over 2 million interactions between almost 10,000 users and more than 1,000 merchants over a period of 14 months, from January 2019 to February 2020.…”

“…The choice of datasets for our research question is limited, as the datasets need to have timestamps included so that the data can be spit and evaluated based on time. We also filter the dataset to only include users and items with at least one interaction at each period, following prior works [28]. Statistical information on the raw datasets is summarized in Table 2.…”

“…Various new methods are constantly being developed in order to improve user recommendations with deep learning techniques [35,45,61], memory-based [51] and latent factor-based methods [11,29,46], or even reinforcement learning [1,36]. On the other hand, despite all this progress, the RecSys evaluation protocol is still an open question due to various possible data splitting strategies and data preparation approaches [12,28,52]. In this way, in order to unlock the full potential of data-driven approaches, we still require more nuanced evaluation techniques to fully estimate RecSys methods on historical data.…”