ClueWeb22: 10 Billion Web Documents with Rich Information

Abstract

ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academic and industry research, for example, in information systems, retrieval-augmented AI systems, and model pretraining. Compared with earlier ClueWeb corpora, the ClueWeb22 corpus is larger, more varied, of higher-quality, and aligned with the document distributions in commercial web search.Besides raw HTML, ClueWeb22 includes rich information about the web pages provided by industry-standard document understanding systems, including the visual representation of pages rendered by a web browser, parsed HTML structure information from a neural network parser, and pre-processed cleaned document text to lower the barrier to entry. Many of these signals have been widely used in industry but are available to the research community for the first time at this scale. INTRODUCTIONLarge scale web corpora are essential for the research and development of technologies such as information retrieval (IR), natural language processing (NLP), and deep learning. Previous ClueWeb corpora, ClueWeb09 [3] and ClueWeb12 [4],have been the standard web corpora for many research explorations during the last decade (e.g., [7,8,35]), but they are more than ten years old. The web and users have changed, new research frontiers have emerged, and our web corpora must evolve with them.Many important research topics in information retrieval are centered around web search. As one of the most used AI applications, web search presents research challenges in user understanding, relevance modeling, indexing, serving efficiency, and many more. All of them rely on a large scale, realistic web corpus to reflect the data distribution and the realistic challenges of web search. Perhaps due to the lack of updated web corpora, many recent IR benchmarks use the derived MS MARCO corpus [2] which was sampled from the top retrieved passages of a Bing question answering system nearly six year ago, which is very different from the distribution of the web.An emerging usage of web corpora is in retrieval-augmented AI systems, for example, to provide documents that serve as evidence for question answering [5], grounding information to improve language generation [18], and enriched contexts for language models [15]. A large fraction of these systems use Wikipedia as the retrieval corpus, which * The first two authors contributed equally to this research.

NeuralMind-UNICAMP at 2022 TREC NeuCLIR: Large Boring Rerankers for Cross-lingual Retrieval

“…Information Retrieval (IR) has a long and robust history in research, with numerous publicly accessible datasets developed to support its advancement. Some of the most well-known and widely-used ones are MS-MARCO [2], TREC [35,38,5], Common Crawl [32], and ClueWeb22 [24]. In response to the challenges of IR, various models and methods have been proposed, utilizing both classic ones such as Vector Space Model (VSM) [31], Latent Semantic Indexing (LSI) [6] and BM25, as well as more modern transformer-based models, such as RoBERTa [19], BERT [7], and T5 [27].…”

Exploring BERT for Aspect Extraction in Portuguese Language

“…BERTimbau used brWac (Wagner Filho et al, 2018), a 2.7B token dataset obtained from crawling PT-BR websites, while Albertina used a PT-PT filtered version of OS-CAR, together with PT-PT transcripts datasets from the Portuguese and the EU Parliaments (Hajlaoui et al, 2014;Koehn, 2005). Sabiá uses the Portuguese subset of ClueWeb22 (Overwijk et al, 2022). Leveraging filtered massive web crawls such as ClueWeb22 (Overwijk et al, 2022) and OSCAR (Abadji et al, 2022), the Portuguese webarchive (Gomes et al, 2008) (Arquivo.pt), encyclopedic and dialog data, we assemble and contribute with a large and highly-diverse pre-training PT-PT corpus.…”

“…Sabiá uses the Portuguese subset of ClueWeb22 (Overwijk et al, 2022). Leveraging filtered massive web crawls such as ClueWeb22 (Overwijk et al, 2022) and OSCAR (Abadji et al, 2022), the Portuguese webarchive (Gomes et al, 2008) (Arquivo.pt), encyclopedic and dialog data, we assemble and contribute with a large and highly-diverse pre-training PT-PT corpus.…”

“…To gather high-quality, large-scale, PT language resources, we resorted to multiple PT-PT text sources, summarized in Table 1. OSCAR-2201 (Abadji et al, 2022) and ClueWeb-L 22 (Overwijk et al, 2022) are web crawls -they both give us text from blogs, forums, among other websites. The PTWiki 3 provides our model with well-written and reviewed encyclopedic knowledge, in neutral and revised Portuguese text.…”

Materiais arqueológicos da Cividade de Bagunte presentes no Museu de História Natural e da Ciência da Universidade do Porto

“…The datasets identified tend to adopt Common Crawl licenses. It is worth mentioning the case of the Portuguese LLM Glória (Lopes et al, 2024), where the usage of the clueweb22 dataset (Overwijk et al, 2022) as part of the training corpus required Glória's authors to adopt this license as well.…”