GemNN: Gating-enhanced Multi-task Neural Networks with Feature Interaction Learning for CTR Prediction

Fei, Hongliang; Zhang, Jingyuan; Zhou, Xingxuan; Zhao, Junhao; Qi, Xinyang; Li, Ping

doi:10.1145/3404835.3463116

Cited by 33 publications

(11 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The influence of 𝜖 ′ 𝑠 decaying rate on performance has been studied by [11] for channel pruning, and similar to it, the decaying rate of 𝜖 in ( 4) and ( 8) is not much important for offline feature selection, and our experiments found that the final value of 𝜖 between 1𝑒 − 4 and 1𝑒 −8 has no significant effect on the performance of the model. The key component of the gate function (8) for LPFS++ is the second arctangent function part and its balancing factor 𝛼. For simplicity, we will focus on 𝛼.…”

Section: Ablation Studies and Experimental Analysismentioning

confidence: 99%

“…The main limitation for function (8), from the perspective of dimensional analysis, is that it is not scale-invariant. The 𝑥 in the arctangent trigonometric function arctan means that 𝑥 must be dimensionless.…”

Section: Conclusion Limitation and Future Workmentioning

confidence: 99%

“…Click-Through Rate (CTR) prediction, which aims to estimate the probability of a user clicking on an item, has become a crucial task in industrial applications, such as personalized recommendations and online advertising [28,38,39]. In recent years, significant progress has been made due to the development of deep learning [8,10,16,31,35], however, these deep models still require an effective set of feature fields as input. In industry, to accurately characterize user preferences, item characteristics, and contextual environments from different aspects, extensive feature engineering is generally essential for model training, which results in hundreds of feature fields in real-world datasets.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LPFS: Learnable Polarizing Feature Selection for Click-Through Rate Prediction

Guo¹,

Liu²,

Tan³

et al. 2022

Preprint

View full text Add to dashboard Cite

In industry, feature selection is a standard but necessary step to search for an optimal set of informative feature fields for efficient and effective training of deep Click-Through Rate (CTR) models. Most previous works measure the importance of feature fields by using their corresponding continuous weights from the model, then remove the feature fields with small weight values. However, removing many features that correspond to small but not exact zero weights will inevitably hurt model performance and not be friendly to hot-start model training. There is also no theoretical guarantee that the magnitude of weights can represent the importance, thus possibly leading to sub-optimal results if using these methods.To tackle this problem, we propose a novel Learnable Polarizing Feature Selection (LPFS) method using a smoothed-ℓ 0 function in literature. Furthermore, we extend LPFS to LPFS++ by our newly designed smoothed-ℓ 0 -liked function to select a more informative subset of features. LPFS and LPFS++ can be used as gates inserted at the input of the deep network to control the active and inactive state of each feature. When training is finished, some gates are exact zero, while others are around one, which is particularly favored by the practical hot-start training in the industry, due to no damage

show abstract

Section: Ablation Studies and Experimental Analysismentioning

confidence: 99%

Section: Conclusion Limitation and Future Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LPFS: Learnable Polarizing Feature Selection for Click-Through Rate Prediction

Guo¹,

Liu²,

Tan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…MMoE has been selected as the feature extractor for ESCM 2 and its competitors. It would also make sense to replace it with more powerful models such as AITM [23] and GemNN [3]. For fairness consideration, all models are trained for 300k iterations with Adam [10] optimizer and the same set of hyperparameters to make results comparable.…”

Section: Training Protocolmentioning

confidence: 99%

ESCM$^2$: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation

Wang,

Chang,

Liu

et al. 2022

Preprint

View full text Add to dashboard Cite

Accurate estimation of post-click conversion rate is critical for building recommender systems, which has long been confronted with sample selection bias and data sparsity issues. Methods in the Entire Space Multi-task Model (ESMM) family leverage the sequential pattern of user actions, i.e., 𝑖𝑚𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛 → 𝑐𝑙𝑖𝑐𝑘 → 𝑐𝑜𝑛𝑣𝑒𝑟𝑠𝑖𝑜𝑛 to address data sparsity issue. However, they still fail to ensure the unbiasedness of CVR estimates. In this paper, we theoretically demonstrate that ESMM suffers from the following two problems:(1) Inherent Estimation Bias (IEB), where the estimated CVR of ESMM is inherently higher than the ground truth; (2) Potential Independence Priority (PIP) for CTCVR estimation, where there is a risk that the ESMM overlooks the causality from click to conversion. To this end, we devise a principled approach named Entire Space Counterfactual Multi-task Modelling (ESCM 2 ), which employs a counterfactual risk miminizer as a regularizer in ESMM to address both IEB and PIP issues simultaneously. Extensive experiments on offline datasets and online environments demonstrate that our proposed ESCM 2 can largely mitigate the inherent IEB and PIP issues and achieve better performance than baseline models.

show abstract

“…A hierarchical model with micro- and macro-behaviour ( HM 3 ) [ 12 ] is proposed, which utilizes multitask learning and applies the abundant supervisory labels from micro- and macro-behaviours to predict conversion rate (CVR) in a unified framework. Gating-enhanced multitask neural network (Gem-NN) [ 13 ] is proposed to predict CTR in a coarse-to-fine manner, which allows parameter sharing from upper-level tasks to lower-level tasks and introduces a gating mechanism between embedding layers and MLP. The multiple-level sparse sharing model (MSSM) [ 14 ] is proposed to represent feature flexibly and share information among tasks efficiently, which include a field-level sparse connection module (FSCM) and a cell-level sparse sharing module (CSSM).…”

Section: Introductionmentioning

confidence: 99%

An Improved Multitask Learning Model with Matching Network and Its Application in Traditional Chinese Medicine Syndrome Recommendation

Wang

Zhang

et al. 2022

Journal of Healthcare Engineering

View full text Add to dashboard Cite

Multitask learning (MTL) is an open and challenging problem in various real-world applications, such as recommendation systems, natural language processing, and computer vision. The typical way of conducting multitask learning is establishing some global parameter sharing mechanism among all tasks or assigning each task an individual set of parameters with cross-connections between tasks. However, for most existing approaches, the raw features are abstracted step by step, semantic information is mined from input space, and matching relation features are not introduced into the model. To solve the above problems, we propose a novel MMOE-match network to model the matches between medical cases and syndrome elements and introduce the recommendation algorithm into traditional Chinese medicine (TCM) study. Accurate medical record recommendation is significant for intelligent medical treatment. Ranking algorithms can be introduced in multi-TCM scenarios, such as syndrome element recommendation, symptom recommendation, and drug prescription recommendation. The recommendation system includes two main stages: recalling and ranking. The core of recalling and ranking is a two-tower matching network and multitask learning. MMOE-match combines the advantages of recalling and ranking model to design a new network. Furtherly, we try to take the matching network output as the input of multitask learning and compare the matching features designed by the manual. The data show that our model can bring significant positive benefits.

show abstract

GemNN: Gating-enhanced Multi-task Neural Networks with Feature Interaction Learning for CTR Prediction

Cited by 33 publications

References 46 publications

LPFS: Learnable Polarizing Feature Selection for Click-Through Rate Prediction

LPFS: Learnable Polarizing Feature Selection for Click-Through Rate Prediction

ESCM$^2$: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation

An Improved Multitask Learning Model with Matching Network and Its Application in Traditional Chinese Medicine Syndrome Recommendation

Contact Info

Product

Resources

About