Personalized neural language models for real-world query auto completion

Fiorini, Nicolas; Lu, Zhiyong

doi:10.18653/v1/n18-3026

Cited by 16 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent papers have also considered biasing LMs with context beyond the previous sentence and incorporate additional signals such as date-time, geolocation or gender (Ma et al, 2018;Diehl Martinez et al, 2021) or application metadata like dialog act or intent (Masumura et al, 2019;Shenoy et al, 2021;Liu and Lane, 2017). Other sources of context used to bias LMs are personalized content (Jaech and Ostendorf, 2018b;Fiorini and Lu, 2018); conversational turn-taking (Xiong et al, 2018); multi-modal sources (Moriya and Jones, 2018); or even user demographics to suggest fashion suggestions (Denk and Peleteiro Ramallo, 2020).…”

Section: Prior Workmentioning

confidence: 99%

CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

Novotney,

Mukherjee,

Ahmed

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a framework to modularize the training of neural language models that use diverse forms of sentence-external context (including metadata) by eliminating the need to jointly train sentence-external and withinsentence encoders. Our approach, contextual universal embeddings (CUE), trains LMs on one set of context, such as date and author, and adapts to novel metadata types, such as article title, or previous sentence. The model consists of a pretrained neural sentence LM, a BERT-based context encoder, and a masked transformer decoder that estimates LM probabilities using sentence-internal and sentenceexternal information. When context or metadata are unavailable, our model learns to combine contextual and sentence-internal information using noisy oracle unigram embeddings as a proxy. Real contextual information can be introduced later and used to adapt a small number of parameters that map contextual data into the decoder's embedding space. We validate the CUE framework on a NYTimes text corpus with multiple metadata types, for which the LM perplexity can be lowered from 36.6 to 27.4 by conditioning on context. Bootstrapping a contextual LM with only a subset of the context/metadata during training retains 85% of the achievable gain. Training the model initially with proxy context retains 67% of the perplexity gain after adapting to real context. Furthermore, we can swap one type of pretrained sentence LM for another without retraining the context encoders, by only adapting the decoder model. Overall, we obtain a modular framework that allows incremental, scalable training of context-enhanced LMs.

show abstract

Section: Prior Workmentioning

confidence: 99%

CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

Novotney,

Mukherjee,

Ahmed

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other approaches further improve over this by ranking suggestions based on previous queries, user profile, time-sensitivity, or coherence with the typed prefix [5,34,36,39]. Recent work also leverages deep learning models to generate additional features such as query likelihood using language models [30], personalized language models [12,14], or previous queries [36] to rank queries using a learning to rank framework [41]. However, these models typically rely on heuristic methods to generate a fixed number of candidates for all popular prefixes.…”

Section: Related Workmentioning

confidence: 99%

Session-Aware Query Auto-completion using Extreme Multi-Label Ranking

Yadav

Sen

Hill

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Query auto-completion (QAC) is a fundamental feature in search engines where the task is to suggest plausible completions of a prefix typed in the search bar. Previous queries in the user session can provide useful context for the user's intent and can be leveraged to suggest auto-completions that are more relevant while adhering to the user's prefix. Such session-aware QACs can be generated by recent sequence-to-sequence deep learning models; however, these generative approaches often do not meet the stringent latency requirements of responding to each user keystroke. Moreover, these generative approaches pose the risk of showing nonsensical queries. One can pre-compute a relatively small subset of relevant queries for common prefixes and rank them based on the context. However, such an approach fails when no relevant queries for the current context are present in the pre-computed set.

show abstract

“…Size Publicly Available Citations AOL 16M queries 3M sessions Yes [38], [21], [11], [1], [12], [37], [40], [30], [31], [8], [35], [7], [6], [5], [19], [34], [10], [15], [20], [11], [13] MS MARCO 1M queries Yes [1], [40], [6], [5] Yahoo Search Engine 4M queries 549K sessions No [25] Tencent website 160M queries No [17], [16] "Baidu Knows" Website 85K pairs of (question, best answer) No [27] later switches to searching about dogs, which shows gradual topic drift revolving around the abstract concept of animals. For this reason, we believe that a gold standard dataset of queries is required that would not rely on the weak assumption of gradual query improvement within the same session.…”

Section: Dataset Namementioning

confidence: 99%

ReQue

Tamannaee

Fani

Zarrinkalam

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

In this paper, we implement and publicly share a configurable software workflow and a collection of gold standard datasets for training and evaluating supervised query refinement methods. Existing datasets such as AOL and MS MARCO, which have been extensively used in the literature, are based on the weak assumption that users' input queries improve gradually within a search session, i.e., the last query where the user ends her information seeking session is the best reconstructed version of her initial query. In practice, such an assumption is not necessarily accurate for a variety of reasons, e.g., topic drift. The objective of our work is to enable researchers to build gold standard query refinement datasets without having to rely on such weak assumptions. Our software workflow, which generates such gold standard query datasets, takes three inputs: (1) a dataset of queries along with their associated relevance judgements (e.g. TREC topics), (2) an information retrieval method (e.g., BM25), and (3) an evaluation metric (e.g., MAP), and outputs a gold standard dataset. The produced gold standard dataset includes a list of revised queries for each query in the input dataset, each of which effectively improves the performance of the specified retrieval method in terms of the desirable evaluation metric. Since our workflow can be used to generate gold standard datasets for any input query set, in this paper, we have generated and publicly shared gold standard datasets for TREC queries associated with Robust04, Gov2, ClueWeb09, and ClueWeb12. The source code of our software workflow, the generated gold datasets, and benchmark results for three state-of-the-art supervised query refinement methods over these datasets are made publicly available for reproducibility purposes.

show abstract

Personalized neural language models for real-world query auto completion

Cited by 16 publications

References 18 publications

CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

Session-Aware Query Auto-completion using Extreme Multi-Label Ranking

ReQue

Contact Info

Product

Resources

About