This paper studies keyphrase extraction in real-world scenarios where documents are from diverse domains and have variant content quality. We curate and release OpenKP, a large scale open domain keyphrase extraction dataset with near one hundred thousand web documents and expert keyphrase annotations. To handle the variations of domain and content quality, we develop BLING-KPE, a neural keyphrase extraction model that goes beyond language understanding using visual presentations of documents and weak supervision from search queries. Experimental results on OpenKP confirm the effectiveness of BLING-KPE and the contributions of its neural architecture, visual features, and search log weak supervision. Zero-shot evaluations on DUC-2001 demonstrate the improved generalization ability of learning from the open domain data compared to a specific domain.

Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space. Autoencoder-based language models are appealing in dense retrieval as they train the encoder to output highquality embedding that can reconstruct the input texts. However, in this paper, we provide theoretical analyses and show empirically that an autoencoder language model with a low reconstruction loss may not provide good sequence representations because the decoder may take shortcuts by exploiting language patterns. To address this, we propose a new selflearning method that pre-trains the autoencoder using a weak decoder, with restricted capacity and attention flexibility to push the encoder to provide better text representations. Our experiments on web search, news recommendation, and open domain question answering show that our pre-trained model significantly boosts the effectiveness and few-shot ability of dense retrieval models. Our code is available at https://github.com/microsoft/ SEED-Encoder/.

Abstract. This paper presents WikiTranslate, a system which performs query translation for cross-lingual information retrieval (CLIR) using only Wikipedia to obtain translations. Queries are mapped to Wikipedia concepts and the corresponding translations of these concepts in the target language are used to create the final query. WikiTranslate is evaluated by searching with topics formulated in Dutch, French and Spanish in an English data collection. The system achieved a performance of 67% compared to the monolingual baseline.

No abstract

ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academic and industry research, for example, in information systems, retrieval-augmented AI systems, and model pretraining. Compared with earlier ClueWeb corpora, the ClueWeb22 corpus is larger, more varied, of higher-quality, and aligned with the document distributions in commercial web search.Besides raw HTML, ClueWeb22 includes rich information about the web pages provided by industry-standard document understanding systems, including the visual representation of pages rendered by a web browser, parsed HTML structure information from a neural network parser, and pre-processed cleaned document text to lower the barrier to entry. Many of these signals have been widely used in industry but are available to the research community for the first time at this scale. INTRODUCTIONLarge scale web corpora are essential for the research and development of technologies such as information retrieval (IR), natural language processing (NLP), and deep learning. Previous ClueWeb corpora, ClueWeb09 [3] and ClueWeb12 [4],have been the standard web corpora for many research explorations during the last decade (e.g., [7,8,35]), but they are more than ten years old. The web and users have changed, new research frontiers have emerged, and our web corpora must evolve with them.Many important research topics in information retrieval are centered around web search. As one of the most used AI applications, web search presents research challenges in user understanding, relevance modeling, indexing, serving efficiency, and many more. All of them rely on a large scale, realistic web corpus to reflect the data distribution and the realistic challenges of web search. Perhaps due to the lack of updated web corpora, many recent IR benchmarks use the derived MS MARCO corpus [2] which was sampled from the top retrieved passages of a Bing question answering system nearly six year ago, which is very different from the distribution of the web.An emerging usage of web corpora is in retrieval-augmented AI systems, for example, to provide documents that serve as evidence for question answering [5], grounding information to improve language generation [18], and enriched contexts for language models [15]. A large fraction of these systems use Wikipedia as the retrieval corpus, which * The first two authors contributed equally to this research.