Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

Dong, Qi; Liu, Yiding; Cheng, Suqi; Wang, Shuaiqiang; Cheng, Zhicong; Niu, Shuzi; Yin, Daqiang

doi:10.48550/arxiv.2204.11673

Cited by 3 publications

(3 citation statements)

References 37 publications

(59 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers have devoted substantial efforts to knowledgeenhanced techniques due to the superiority of diversified knowledge from different domain [17]- [19]. Recently, considerable knowledge-enhanced RS have been proposed for accurate interest modeling [20]- [24].…”

Section: B Knowledge-enhanced Techniquesmentioning

confidence: 99%

Social4Rec: Distilling User Preference from Social Graph for Video Recommendation in Tencent

Xiao¹,

Dai²,

Dong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Despite recommender systems play a key role in network content platforms, mining the user's interests is still a significant challenge. Existing works predict the user interest by utilizing user behaviors, i.e., clicks, views, etc., but current solutions are ineffective when users perform unsettled activities. The latter ones involve new users, which have few activities of any kind, and sparse users who have low-frequency behaviors. We uniformly describe both these user-types as "cold users", which are very common but often neglected in network content platforms. To address this issue, we enhance the representation of the user interest by combining his social interest, e.g., friendship, following bloggers, interest groups, etc., with the activity behaviors. Thus, in this work, we present a novel algorithm entitled SocialNet, which adopts a two-stage method to progressively extract the coarse-grained and fine-grained social interest. Our technique then concatenates SocialNet's output with the original user representation to get the final user representation that combines behavior interests and social interests. Offline experiments on Tencent video's recommender system demonstrate the superiority over the baseline behavior-based model. The online experiment also shows a significant performance improvement in clicks and view time in the real-world recommendation system.

show abstract

Section: B Knowledge-enhanced Techniquesmentioning

confidence: 99%

Social4Rec: Distilling User Preference from Social Graph for Video Recommendation in Tencent

Xiao¹,

Dai²,

Dong³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Pretrained language models, such as BERT [9] and T5 [51], have demonstrated superior performance on many natural language tasks including information retrieval [13,38,66,74]. PLMs are usually pretrained on general corpora and then finetuned on the target corpus.…”

Section: Pretrained Language Models For Searchmentioning

confidence: 99%

Pre-trained Language Model for Web-scale Retrieval in Baidu Search

Liu

Cheng

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Search engine plays a crucial role in satisfying users' diverse information needs. Recently, Pretrained Language Models (PLMs) based text ranking models have achieved huge success in web search. However, many state-of-the-art text ranking approaches only focus on core relevance while ignoring other dimensions that contribute to user satisfaction, e.g., document quality, recency, authority, etc. In this work, we focus on ranking user satisfaction rather than relevance in web search, and propose a PLM-based framework, namely SAT-Ranker, which comprehensively models different dimensions of user satisfaction in a unified manner. In particular, we leverage the capacities of PLMs on both textual and numerical inputs, and apply a multi-field input that modularizes each dimension of user satisfaction as an input field. Overall, SAT-Ranker is an effective, extensible, and data-centric framework that has huge potential for industrial applications. On rigorous offline and online experiments, SAT-Ranker obtains remarkable gains on various evaluation sets targeting different dimensions of user satisfaction. It is now fully deployed online to improve the usability of our search engine. CCS CONCEPTS• Information systems → Language models; Learning to rank.

show abstract

“…Hence, IR researchers propose to utilize pre-trained language models (PLM), i.e., large-scale neural models trained without supervised data for language understanding, to conduct effective retrieval [13,17,36,43]. Previous studies [7,10,16,42] have shown that PLM such as BERT [9] and RoBERTa [18] significantly outperform existing neural retrieval models on passage and document retrieval datasets like MS MARCO and TREC DL in both zero-shot and few-shot settings [8,25,35].…”

Section: Introductionmentioning

confidence: 99%

Building Chinese interdisciplinary research centers : the case of Tsinghua University

Li¹,

李蓉暉²

View full text Add to dashboard Cite

Graph machine learning has been extensively studied in both academia and industry. However, in the literature, most existing graph machine learning models are designed to conduct training with data samples in a random order, which may suffer from suboptimal performance due to ignoring the importance of different graph data samples and their training orders for the model optimization status. To tackle this critical problem, curriculum graph machine learning (Graph CL), which integrates the strength of graph machine learning and curriculum learning, arises and attracts an increasing amount of attention from the research community. Therefore, in this paper, we comprehensively overview approaches on Graph CL and present a detailed survey of recent advances in this direction. Specifically, we first discuss the key challenges of Graph CL and provide its formal problem definition. Then, we categorize and summarize existing methods into three classes based on three kinds of graph machine learning tasks, i.e., node-level, linklevel, and graph-level tasks. Finally, we share our thoughts on future research directions. To the best of our knowledge, this paper is the first survey for curriculum graph machine learning.

show abstract

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

Cited by 3 publications

References 37 publications

Social4Rec: Distilling User Preference from Social Graph for Video Recommendation in Tencent

Social4Rec: Distilling User Preference from Social Graph for Video Recommendation in Tencent

Pre-trained Language Model for Web-scale Retrieval in Baidu Search

Building Chinese interdisciplinary research centers : the case of Tsinghua University

Contact Info

Product

Resources

About