Product-Based Neural Networks for User Response Prediction over Multi-Field Categorical Data

Abstract

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.Many machine learning models are leveraged or proposed to work on such a problem, including linear models, latent vector-based models, tree models, and DNN-based models. Linear models, such as Logistic Regression (LR) [25] and Bayesian Probit Regression [14], are easy to implement and with high efficiency. A typical latent vector-based model is Factorization Machine (FM) [36]. FM uses weights and latent vectors to represent categories. According to their parametric representations, LR has a linear feature extractor, and FM has a bi-linear 2 feature extractor. The prediction of LR and FM are simply based on the sum over weights, thus their classifiers are linear. FM works well on sparse data, and inspires a lot of extensions, including Field-aware FM (FFM) [21]. FFM introduces field-aware latent vectors, which gain FFM higher capacity and better performance. However, FFM is restricted by space complexity. Inspired by FFM, we find a coupled gradient issue of latent vector-based models and refine feature interactions 3 as field-aware feature interactions. To solve this issue as well as saving memory, we propose kernel product methods and derive Kernel FM (KFM) and Network in FM (NIFM).Trees and DNNs are potent function approximators. Tree models, such as Gradient Boosting Decision Tree (GBDT) [6], are popular in various data science contests as well as industrial applications. GBDT explores very high order feature combinations in a non-parametric way, yet its exploration ability is restricted when feature space becomes extremely high-dimensional and sparse. DNN has also been preliminarily studied in information system literature [8,33,40,51]. In [51], FM supported Neura...

An end-to-end neighborhood-based interaction model for knowledge-enhanced recommendation

“…The first is feature-based, which regards side information as plain features and concatenates those features with user/item IDs as model input, including Matrix factorization models [13,16], DNN models [6,19,20], etc. Feature-based models highly rely on manual feature engineering to extract structural information, which is not end-to-end and less efficient.…”

Personalized re-ranking for recommendation

“…Inspired by the work [13], we employ the hidden vector before the sigmoid layer as the personalized vector pv i (in Figure 1(c)) that feeds into our PRM model. Figure 1(c) shows one possible architecture of the pre-trained model, other general models such as FM [25], FFM [23], DeepFM [16], DCN [30], FNN [36] and PNN [24] can also be used as alternatives to generate PV .…”

“…(2) Equipped with long term and generic user embedding, our PRM model is able to learn be er user-speci c encoding function which can more precisely capture mutual in uences of item-pairs for each user. Note that the architecture of the pre-trained model is not highly coupled with our PRM model, other general models [16,23,24,30,36] can also be used as alternatives to generate PV .…”

Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction

“…Click-Through Rate (CTR) is a crucial task for recommender systems, which estimates the probability of a user to click on a given item [9,23]. In an online advertising application, which is a billiondollar scenario, the ranking strategy of candidate advertisements is by CTR×bid where "bid" is the profit that the system receives once the advertisement is clicked on.…”

“…In an online advertising application, which is a billiondollar scenario, the ranking strategy of candidate advertisements is by CTR×bid where "bid" is the profit that the system receives once the advertisement is clicked on. In such applications, the performance of CTR prediction models [9,23,40] is one of the core factors determining system's revenue.…”