Architecture and Operation Adaptive Network for Online Recommendations

DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction

“…Feature interactions are learned by utilizing some functions to model the relationship among the embeddings. Multiple studies [6,13,18,20,22,37,39] have shown that a better design of the interaction part can generate significant lift on the prediction accuracy, which motivates a variety of designs from different perspectives. Feature Interaction.…”

“…To automatically learn which feature interactions are essential, AutoFIS [22] introduces a gate (in open or closed status) for each feature interaction to control whether its output should be passed to the next layer. Similarly, GIN [18] relies on a second-stage re-training to prune unimportant feature interactions. AIM [51] further improves AutoFIS through relaxing discrete selection of open gates to be continuous parameters that can be jointly trained with model parameters and allowing for dedicated embedding size for features of different importance.…”

“…In the end, it is worth mentioning the architectures closest to DHEN from prior studies are GIN [18] and AutoInt [37]. Compared to them, our study has salient differences: although GIN and Au-toInt also uses more than one interaction module (the multi-head attention structure in GIN and the multi-branch structure in Au-toInt), each head is still homogeneous, which cannot complement non-overlapping information and capture the correlation of different modules as well as the hierarchy.…”

“…Feature interactions are learned by utilizing functions to model the relationship among the embeddings. Multiple studies [6,13,18,20,22,37,39] have shown that a better design of the interaction part can lift the prediction accuracy significantly over real-world applications. This has motivated a variety of studies that evolves from capturing low-order interactions towards high-order ones [7, 11, 13, 18-20, 22, 23, 29, 30, 34, 40, 43, 43, 48, 51].…”

“…This indicates that different interaction modules intended to capture the same order of interaction have different strengths over different datasets, and moreover the root cause might be the information captured by them isn't overlapping. In the meantime, as shown by the experimental parts in those studies, the performance improvement obtained through learning higher-order interaction (often realized by stacking more interaction layers) sometimes has a negative effect (Figure 3 in DCN [39], Figure 7(a) in xDeepFM [20], Figure 4 in InterHAt [19], Table 2 in xDeepInt [43], and Figure 3(a) in GIN [18], to name a few). This is counter-intuitive as according to theory, capturing higher-order interaction should lead to better or at least neutral performance.…”